[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:45:05 dist-test-slave-twl6 krb5kdc[7367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:45:05 dist-test-slave-twl6 krb5kdc[7367](info): set up 2 sockets
Sep 28 06:45:05 dist-test-slave-twl6 krb5kdc[7367](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 06:45:06.584211 7360 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:45:06 dist-test-slave-twl6 krb5kdc[7367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:45:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:45:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:45:06.703147 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:40635
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:34291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:40635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:06.814839 7380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:06.815102 7380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:06.815146 7380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:06.819453 7380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:45:06.819502 7380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:06.819523 7380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:06.819541 7380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:45:06.819558 7380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:45:06.824805 7380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:34291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:40635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:40635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:06.825757 7380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:06.826732 7380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:06.835597 7385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:06.835628 7388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:06.835718 7380 server_base.cc:1034] running on GCE node
W20260928 06:45:06.835646 7386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:06.836412 7380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:06.837361 7380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:06.838542 7380 hybrid_clock.cc:648] HybridClock initialized: now 1790577906838524 us; error 28 us; skew 500 ppm
Sep 28 06:45:06 dist-test-slave-twl6 krb5kdc[7367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577906, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:06.841212 7380 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:45:06.842123 7380 webserver.cc:460] Webserver started at http://127.7.48.62:36431/ using document root <none> and password file <none>
I20260928 06:45:06.842706 7380 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:06.842794 7380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:06.843036 7380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:06.845022 7380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "942bf4c485e74a8684c4e04f65b949fb"
format_stamp: "Formatted at 2026-09-28 06:45:06 on dist-test-slave-twl6"
server_key: "bc47fd4a512becc5a2ae3d3703a1fbf9"
server_key_iv: "aa20e4672481034688f7a7b268985647"
server_key_version: "encryptionkey@0"
I20260928 06:45:06.845580 7380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "942bf4c485e74a8684c4e04f65b949fb"
format_stamp: "Formatted at 2026-09-28 06:45:06 on dist-test-slave-twl6"
server_key: "bc47fd4a512becc5a2ae3d3703a1fbf9"
server_key_iv: "aa20e4672481034688f7a7b268985647"
server_key_version: "encryptionkey@0"
I20260928 06:45:06.849131 7380 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:45:06.851408 7395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:06.852612 7380 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:45:06.852744 7380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "942bf4c485e74a8684c4e04f65b949fb"
format_stamp: "Formatted at 2026-09-28 06:45:06 on dist-test-slave-twl6"
server_key: "bc47fd4a512becc5a2ae3d3703a1fbf9"
server_key_iv: "aa20e4672481034688f7a7b268985647"
server_key_version: "encryptionkey@0"
I20260928 06:45:06.852849 7380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:06.867791 7380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:06.870376 7380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:06.870568 7380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:06.877264 7380 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:40635
I20260928 06:45:06.877288 7447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:40635 every 8 connection(s)
I20260928 06:45:06.878091 7380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:45:06.878808 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 7380
I20260928 06:45:06.878955 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:45:06.880175 7360 external_mini_cluster.cc:1428] Setting key 966dd7607b01c6ef8884171d298bd1d3
I20260928 06:45:06.880779 7448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:06.886792 7448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 942bf4c485e74a8684c4e04f65b949fb: Bootstrap starting.
I20260928 06:45:06.888851 7448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 942bf4c485e74a8684c4e04f65b949fb: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:06.889535 7448 log.cc:824] T 00000000000000000000000000000000 P 942bf4c485e74a8684c4e04f65b949fb: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:06.891500 7448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 942bf4c485e74a8684c4e04f65b949fb: No bootstrap required, opened a new log
I20260928 06:45:06.895473 7448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 942bf4c485e74a8684c4e04f65b949fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942bf4c485e74a8684c4e04f65b949fb" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 40635 } }
I20260928 06:45:06.895699 7448 raft_consensus.cc:374] T 00000000000000000000000000000000 P 942bf4c485e74a8684c4e04f65b949fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:06.895779 7448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 942bf4c485e74a8684c4e04f65b949fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 942bf4c485e74a8684c4e04f65b949fb, State: Initialized, Role: FOLLOWER
I20260928 06:45:06.896262 7448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 942bf4c485e74a8684c4e04f65b949fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942bf4c485e74a8684c4e04f65b949fb" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 40635 } }
Sep 28 06:45:06 dist-test-slave-twl6 krb5kdc[7367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:06.896399 7448 raft_consensus.cc:388] T 00000000000000000000000000000000 P 942bf4c485e74a8684c4e04f65b949fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:45:06.896459 7448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 942bf4c485e74a8684c4e04f65b949fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:45:06.896564 7448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 942bf4c485e74a8684c4e04f65b949fb [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:06.897500 7448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 942bf4c485e74a8684c4e04f65b949fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942bf4c485e74a8684c4e04f65b949fb" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 40635 } }
I20260928 06:45:06.897879 7448 leader_election.cc:302] T 00000000000000000000000000000000 P 942bf4c485e74a8684c4e04f65b949fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 942bf4c485e74a8684c4e04f65b949fb; no voters:
I20260928 06:45:06.898254 7448 leader_election.cc:288] T 00000000000000000000000000000000 P 942bf4c485e74a8684c4e04f65b949fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:45:06.898373 7453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 942bf4c485e74a8684c4e04f65b949fb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:06.898602 7453 raft_consensus.cc:686] T 00000000000000000000000000000000 P 942bf4c485e74a8684c4e04f65b949fb [term 1 LEADER]: Becoming Leader. State: Replica: 942bf4c485e74a8684c4e04f65b949fb, State: Running, Role: LEADER
I20260928 06:45:06.898891 7453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 942bf4c485e74a8684c4e04f65b949fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942bf4c485e74a8684c4e04f65b949fb" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 40635 } }
I20260928 06:45:06.899493 7448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 942bf4c485e74a8684c4e04f65b949fb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:45:06.900426 7454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 942bf4c485e74a8684c4e04f65b949fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "942bf4c485e74a8684c4e04f65b949fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942bf4c485e74a8684c4e04f65b949fb" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 40635 } } }
I20260928 06:45:06.900568 7454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 942bf4c485e74a8684c4e04f65b949fb [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:06.900946 7462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:45:06.901248 7455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 942bf4c485e74a8684c4e04f65b949fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 942bf4c485e74a8684c4e04f65b949fb. Latest consensus state: current_term: 1 leader_uuid: "942bf4c485e74a8684c4e04f65b949fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942bf4c485e74a8684c4e04f65b949fb" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 40635 } } }
I20260928 06:45:06.901360 7455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 942bf4c485e74a8684c4e04f65b949fb [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:06.903784 7450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:06.883093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42054 (local address 127.7.48.62:40635)
0928 06:45:06.883477 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:06.883491 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:06.884579 (+ 1088us) server_negotiation.cc:407] Connection header received
0928 06:45:06.887343 (+ 2764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:06.887356 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:06.887613 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:06.887911 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:06.889946 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:06.891298 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:06.892515 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:06.893202 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:06.893360 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:06.897193 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:06.897203 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:06.897219 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:06.897248 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:06.899226 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:06.899821 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:06.899827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:06.899832 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:06.900013 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:06.900420 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:06.900425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:06.900426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:06.900949 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:06.901138 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:06.902965 (+ 1827us) server_negotiation.cc:299] Negotiation successful
0928 06:45:06.903163 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":122,"threads_started":1}
I20260928 06:45:06.904245 7462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:45:06.909523 7462 catalog_manager.cc:1347] Generated new cluster ID: d0ac8d6ea7d341648933a2d0785bece2
I20260928 06:45:06.909590 7462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:45:06.929247 7462 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:45:06.930197 7462 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:45:06.938124 7462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 942bf4c485e74a8684c4e04f65b949fb: Generated new TSK 0
I20260928 06:45:06.938776 7462 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
I20260928 06:45:06.970788 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:40635
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:34291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:07.079947 7474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:07.080178 7474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:07.080219 7474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:07.084167 7474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:07.084215 7474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:07.084309 7474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:45:07.089272 7474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:34291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:40635
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:07.090266 7474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:07.091280 7474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:07.100591 7480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:07.100591 7482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:07.100608 7479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:07.100963 7474 server_base.cc:1034] running on GCE node
I20260928 06:45:07.101418 7474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:07.101962 7474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:07.103133 7474 hybrid_clock.cc:648] HybridClock initialized: now 1790577907103122 us; error 25 us; skew 500 ppm
Sep 28 06:45:07 dist-test-slave-twl6 krb5kdc[7367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577907, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:07.105774 7474 init.cc:377] Logged in from keytab as kudu/127.7.48.1@KRBTEST.COM (short username kudu)
I20260928 06:45:07.106588 7474 webserver.cc:460] Webserver started at http://127.7.48.1:44997/ using document root <none> and password file <none>
I20260928 06:45:07.107151 7474 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:07.107240 7474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:07.107470 7474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:07.109555 7474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "eb03a216ee42440289ddc39967b6cfae"
format_stamp: "Formatted at 2026-09-28 06:45:07 on dist-test-slave-twl6"
server_key: "9767605e05462c7556e8baaae9fd44ca"
server_key_iv: "50067c09a637e15f73646b299c256d79"
server_key_version: "encryptionkey@0"
I20260928 06:45:07.110119 7474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "eb03a216ee42440289ddc39967b6cfae"
format_stamp: "Formatted at 2026-09-28 06:45:07 on dist-test-slave-twl6"
server_key: "9767605e05462c7556e8baaae9fd44ca"
server_key_iv: "50067c09a637e15f73646b299c256d79"
server_key_version: "encryptionkey@0"
I20260928 06:45:07.113823 7474 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 06:45:07.116338 7489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:07.117486 7474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:45:07.117616 7474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "eb03a216ee42440289ddc39967b6cfae"
format_stamp: "Formatted at 2026-09-28 06:45:07 on dist-test-slave-twl6"
server_key: "9767605e05462c7556e8baaae9fd44ca"
server_key_iv: "50067c09a637e15f73646b299c256d79"
server_key_version: "encryptionkey@0"
I20260928 06:45:07.117720 7474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:07.133081 7474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:07.135766 7474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:07.135989 7474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:07.136574 7474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:07.137406 7474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:07.137477 7474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:07.137545 7474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:07.137593 7474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:07.147523 7474 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:36025
I20260928 06:45:07.147567 7602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:36025 every 8 connection(s)
I20260928 06:45:07.148599 7474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:45:07.156903 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 7474
I20260928 06:45:07.156997 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
I20260928 06:45:07.157245 7360 external_mini_cluster.cc:1428] Setting key bd4d4a742f6c065f7cc29080c3d76ee0
Sep 28 06:45:07 dist-test-slave-twl6 krb5kdc[7367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577907, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:07.162461 7450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:07.150433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:49839 (local address 127.7.48.62:40635)
0928 06:45:07.150568 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:07.150573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:07.151418 (+ 845us) server_negotiation.cc:407] Connection header received
0928 06:45:07.152772 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:07.152774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:07.152822 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:07.152899 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:07.154032 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:07.154555 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:07.155693 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:07.156020 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:07.156143 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:07.158846 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:07.158850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:07.158852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:07.158874 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:07.160401 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:07.160919 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:07.160924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:07.160926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:07.161007 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:07.161363 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:07.161367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:07.161369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:07.161551 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:07.161663 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:07.162156 (+ 493us) server_negotiation.cc:299] Negotiation successful
0928 06:45:07.162289 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:45:07.163327 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:07.150830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:40635 (local address 127.7.48.1:49839)
0928 06:45:07.151261 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:07.151303 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:07.152565 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:07.153059 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:07.153066 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:07.153410 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:07.153905 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:07.153919 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:07.154662 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:07.154664 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:07.155536 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:07.155544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:07.156173 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:07.156178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:07.156350 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:07.156990 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:07.157015 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:07.158674 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:07.160521 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:07.160527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:07.160543 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:07.160810 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:07.161105 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:07.161108 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:07.161110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:07.161263 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:07.161655 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:07.161661 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:07.161917 (+ 256us) client_negotiation.cc:769] Sending connection context
0928 06:45:07.162114 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 06:45:07.162497 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":115,"threads_started":1}
I20260928 06:45:07.164810 7603 heartbeater.cc:347] Connected to a master server at 127.7.48.62:40635
I20260928 06:45:07.165055 7603 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:07.165542 7603 heartbeater.cc:511] Master 127.7.48.62:40635 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.48.2@KRBTEST.COM; defaulting to no policy
I20260928 06:45:07.167176 7412 ts_manager.cc:194] Registered new tserver with Master: eb03a216ee42440289ddc39967b6cfae (127.7.48.1:36025)
I20260928 06:45:07.168377 7412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.1@KRBTEST.COM'} at 127.7.48.1:49839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
I20260928 06:45:07.184423 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:40635
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:34291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:07.295879 7608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:07.296211 7608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:07.296295 7608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:07.300339 7608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:07.300475 7608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:07.300566 7608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:45:07.305678 7608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:34291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:40635
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:07.306689 7608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:07.307718 7608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:07.317310 7613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:07.317359 7608 server_base.cc:1034] running on GCE node
W20260928 06:45:07.317304 7616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:07.317310 7614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:07.317983 7608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:07.318544 7608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:07.319681 7608 hybrid_clock.cc:648] HybridClock initialized: now 1790577907319660 us; error 32 us; skew 500 ppm
Sep 28 06:45:07 dist-test-slave-twl6 krb5kdc[7367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577907, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:07.322518 7608 init.cc:377] Logged in from keytab as kudu/127.7.48.2@KRBTEST.COM (short username kudu)
I20260928 06:45:07.323351 7608 webserver.cc:460] Webserver started at http://127.7.48.2:34849/ using document root <none> and password file <none>
I20260928 06:45:07.323966 7608 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:07.324056 7608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:07.324292 7608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:07.326316 7608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "328b57ec2c804bd5b94bf797e1d83dc5"
format_stamp: "Formatted at 2026-09-28 06:45:07 on dist-test-slave-twl6"
server_key: "94fc0de1037a092bf1a12e8093f3ad13"
server_key_iv: "543cc7f72d72c0d2ebbad07ac3d01abe"
server_key_version: "encryptionkey@0"
I20260928 06:45:07.326889 7608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "328b57ec2c804bd5b94bf797e1d83dc5"
format_stamp: "Formatted at 2026-09-28 06:45:07 on dist-test-slave-twl6"
server_key: "94fc0de1037a092bf1a12e8093f3ad13"
server_key_iv: "543cc7f72d72c0d2ebbad07ac3d01abe"
server_key_version: "encryptionkey@0"
I20260928 06:45:07.330788 7608 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 06:45:07.333353 7623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:07.334575 7608 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:45:07.334712 7608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "328b57ec2c804bd5b94bf797e1d83dc5"
format_stamp: "Formatted at 2026-09-28 06:45:07 on dist-test-slave-twl6"
server_key: "94fc0de1037a092bf1a12e8093f3ad13"
server_key_iv: "543cc7f72d72c0d2ebbad07ac3d01abe"
server_key_version: "encryptionkey@0"
I20260928 06:45:07.334821 7608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:07.370961 7608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:07.374189 7608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:07.374398 7608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:07.375000 7608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:07.375854 7608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:07.375981 7608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:07.376055 7608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:07.376108 7608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:07.385206 7608 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:38725
I20260928 06:45:07.385231 7736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:38725 every 8 connection(s)
I20260928 06:45:07.386076 7608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:45:07.390838 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 7608
I20260928 06:45:07.390969 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:45:07.391249 7360 external_mini_cluster.cc:1428] Setting key bed627cb29502301db8b04aab9d98739
Sep 28 06:45:07 dist-test-slave-twl6 krb5kdc[7367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577907, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.48.3@KRBTEST.COM; defaulting to no policy
I20260928 06:45:07.400944 7450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:07.387859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:37913 (local address 127.7.48.62:40635)
0928 06:45:07.388023 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:07.388028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:07.388943 (+ 915us) server_negotiation.cc:407] Connection header received
0928 06:45:07.390247 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:07.390253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:07.390309 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:07.390404 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:07.391979 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:07.392679 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:07.393825 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:07.394053 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:07.394174 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:07.396953 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:07.396956 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:07.396959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:07.396990 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:07.398480 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:07.399014 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:07.399017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:07.399019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:07.399110 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:07.399524 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:07.399563 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:07.399566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:07.399816 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:07.399955 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:07.400510 (+ 555us) server_negotiation.cc:299] Negotiation successful
0928 06:45:07.400699 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 06:45:07.401846 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:07.388330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:40635 (local address 127.7.48.2:37913)
0928 06:45:07.388777 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:07.388831 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:07.390025 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:07.390555 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:07.390563 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:07.391084 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:07.391799 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:07.391816 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:07.392803 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:07.392806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:07.393699 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:07.393708 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:07.394207 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:07.394211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:07.394382 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:07.394978 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:07.394993 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:07.396798 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:07.398621 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:07.398627 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:07.398654 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:07.398904 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:07.399205 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:07.399208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:07.399210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:07.399354 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:07.399958 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:07.399963 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:07.400225 (+ 262us) client_negotiation.cc:769] Sending connection context
0928 06:45:07.400425 (+ 200us) client_negotiation.cc:240] Negotiation successful
0928 06:45:07.400839 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":124,"threads_started":1}
I20260928 06:45:07.402999 7737 heartbeater.cc:347] Connected to a master server at 127.7.48.62:40635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.3@KRBTEST.COM" created.
I20260928 06:45:07.403273 7737 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:07.403811 7737 heartbeater.cc:511] Master 127.7.48.62:40635 requested a full tablet report, sending...
I20260928 06:45:07.404867 7412 ts_manager.cc:194] Registered new tserver with Master: 328b57ec2c804bd5b94bf797e1d83dc5 (127.7.48.2:38725)
I20260928 06:45:07.405545 7412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.2@KRBTEST.COM'} at 127.7.48.2:37913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
I20260928 06:45:07.418581 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:40635
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:34291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:07.529556 7742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:07.529801 7742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:07.529842 7742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:07.533841 7742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:07.533896 7742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:07.533973 7742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:45:07.539295 7742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:34291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:40635
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:07.540333 7742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:07.541299 7742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:07.550925 7747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:07.551041 7742 server_base.cc:1034] running on GCE node
W20260928 06:45:07.550896 7748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:07.550899 7750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:07.551666 7742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:07.552301 7742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:07.553500 7742 hybrid_clock.cc:648] HybridClock initialized: now 1790577907553484 us; error 27 us; skew 500 ppm
Sep 28 06:45:07 dist-test-slave-twl6 krb5kdc[7367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577907, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:07.556311 7742 init.cc:377] Logged in from keytab as kudu/127.7.48.3@KRBTEST.COM (short username kudu)
I20260928 06:45:07.557284 7742 webserver.cc:460] Webserver started at http://127.7.48.3:43151/ using document root <none> and password file <none>
I20260928 06:45:07.557845 7742 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:07.557941 7742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:07.558161 7742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:07.560149 7742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "47b7071e3046471f95b461e807095a2b"
format_stamp: "Formatted at 2026-09-28 06:45:07 on dist-test-slave-twl6"
server_key: "4d9de0437e7b60ae65f520c4fe556690"
server_key_iv: "8eadd2e1b60b0af957530096a4e6e6d3"
server_key_version: "encryptionkey@0"
I20260928 06:45:07.560705 7742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "47b7071e3046471f95b461e807095a2b"
format_stamp: "Formatted at 2026-09-28 06:45:07 on dist-test-slave-twl6"
server_key: "4d9de0437e7b60ae65f520c4fe556690"
server_key_iv: "8eadd2e1b60b0af957530096a4e6e6d3"
server_key_version: "encryptionkey@0"
I20260928 06:45:07.564417 7742 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 06:45:07.566874 7757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:07.568135 7742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:45:07.568265 7742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "47b7071e3046471f95b461e807095a2b"
format_stamp: "Formatted at 2026-09-28 06:45:07 on dist-test-slave-twl6"
server_key: "4d9de0437e7b60ae65f520c4fe556690"
server_key_iv: "8eadd2e1b60b0af957530096a4e6e6d3"
server_key_version: "encryptionkey@0"
I20260928 06:45:07.568377 7742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:07.591471 7742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:07.595021 7742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:07.595261 7742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:07.595944 7742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:07.596854 7742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:07.596936 7742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:07.597033 7742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:07.597083 7742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:07.607134 7742 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:34079
I20260928 06:45:07.607160 7870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:34079 every 8 connection(s)
I20260928 06:45:07.608139 7742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:45:07.614707 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 7742
I20260928 06:45:07.614804 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:45:07.615072 7360 external_mini_cluster.cc:1428] Setting key 67b7ca6954514a844fdf0aeed47f4cba
Sep 28 06:45:07 dist-test-slave-twl6 krb5kdc[7367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577907, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:07.622068 7450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:07.609965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:56945 (local address 127.7.48.62:40635)
0928 06:45:07.610147 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:07.610152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:07.611031 (+ 879us) server_negotiation.cc:407] Connection header received
0928 06:45:07.612409 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:07.612412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:07.612461 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:07.612542 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:07.613707 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:07.614241 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:07.615448 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:07.615784 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:07.615914 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:07.618735 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:07.618743 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:07.618745 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:07.618770 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:07.620007 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:07.620515 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:07.620518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:07.620520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:07.620598 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:07.620946 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:07.620949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:07.620951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:07.621186 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:07.621295 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:07.621785 (+ 490us) server_negotiation.cc:299] Negotiation successful
0928 06:45:07.621902 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 06:45:07.622925 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:07.610390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:40635 (local address 127.7.48.3:56945)
0928 06:45:07.610853 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:07.610895 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:07.612207 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:07.612709 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:07.612716 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:07.613070 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:07.613565 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:07.613578 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:07.614348 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:07.614350 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:07.615327 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:07.615337 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:07.616028 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:07.616035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:07.616243 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:07.616886 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:07.616906 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:07.618568 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:07.620119 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:07.620125 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:07.620139 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:07.620402 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:07.620695 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:07.620699 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:07.620701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:07.620844 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:07.621295 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:07.621302 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:07.621543 (+ 241us) client_negotiation.cc:769] Sending connection context
0928 06:45:07.621730 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 06:45:07.622056 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":137,"threads_started":1}
I20260928 06:45:07.624037 7871 heartbeater.cc:347] Connected to a master server at 127.7.48.62:40635
I20260928 06:45:07.624284 7871 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:07.624747 7871 heartbeater.cc:511] Master 127.7.48.62:40635 requested a full tablet report, sending...
I20260928 06:45:07.625705 7412 ts_manager.cc:194] Registered new tserver with Master: 47b7071e3046471f95b461e807095a2b (127.7.48.3:34079)
I20260928 06:45:07.626210 7412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.3@KRBTEST.COM'} at 127.7.48.3:56945
I20260928 06:45:07.629258 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:45:07 dist-test-slave-twl6 krb5kdc[7367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577907, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:45:07 dist-test-slave-twl6 krb5kdc[7367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577907, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.48.1@KRBTEST.COM
I20260928 06:45:07.653844 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:07.645058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57264 (local address 127.7.48.1:36025)
0928 06:45:07.645385 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:07.645402 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:07.645425 (+ 23us) server_negotiation.cc:407] Connection header received
0928 06:45:07.645475 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:07.645481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:07.645674 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:07.645798 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:07.646377 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:07.647100 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:07.647859 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:07.648281 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:07.648365 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:07.650549 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:07.650557 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:07.650569 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:07.650590 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:07.652302 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:07.652647 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:07.652651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:07.652652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:07.652723 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:07.652933 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:07.652935 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:07.652936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:07.653154 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:07.653301 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:07.653453 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 06:45:07.653643 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":95,"threads_started":1}
W20260928 06:45:07.654811 7517 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57264
Sep 28 06:45:07 dist-test-slave-twl6 krb5kdc[7367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:45:07 dist-test-slave-twl6 krb5kdc[7367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.1@KRBTEST.COM
I20260928 06:45:07.679548 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:07.671598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57272 (local address 127.7.48.1:36025)
0928 06:45:07.671786 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:07.671791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:07.671896 (+ 105us) server_negotiation.cc:407] Connection header received
0928 06:45:07.672052 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:07.672054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:07.672096 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:07.672187 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:07.672865 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:07.673397 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:07.674186 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:07.674404 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:07.674478 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:07.676749 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:07.676751 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:07.676753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:07.676775 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:07.678176 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:07.678512 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:07.678514 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:07.678515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:07.678585 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:07.678807 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:07.678809 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:07.678810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:07.678954 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:07.679043 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:07.679225 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 06:45:07.679314 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 06:45:07.680558 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 7474
I20260928 06:45:07.686777 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 7608
I20260928 06:45:07.692340 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 7742
I20260928 06:45:07.697744 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 7380
2026-09-28T06:45:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2165 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:45:07 dist-test-slave-twl6 krb5kdc[7885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:45:07 dist-test-slave-twl6 krb5kdc[7885](info): set up 2 sockets
Sep 28 06:45:07 dist-test-slave-twl6 krb5kdc[7885](info): commencing operation
krb5kdc: starting...
W20260928 06:45:08.727994 7360 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:45:08 dist-test-slave-twl6 krb5kdc[7885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:45:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:45:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:45:08.846700 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:46773
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:36505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:46773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:08.955725 7898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:08.955998 7898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:08.956037 7898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:08.959924 7898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:45:08.959975 7898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:08.959995 7898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:08.960012 7898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:45:08.960027 7898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:45:08.965216 7898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:36505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:46773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:46773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:08.966251 7898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:08.967283 7898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:08.975775 7903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:08.975831 7904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:08.975844 7906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:08.976043 7898 server_base.cc:1034] running on GCE node
I20260928 06:45:08.976712 7898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:08.977809 7898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:08.978997 7898 hybrid_clock.cc:648] HybridClock initialized: now 1790577908978979 us; error 31 us; skew 500 ppm
Sep 28 06:45:08 dist-test-slave-twl6 krb5kdc[7885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577908, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:08.981747 7898 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:45:08.982625 7898 webserver.cc:460] Webserver started at http://127.7.48.62:36855/ using document root <none> and password file <none>
I20260928 06:45:08.983137 7898 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:08.983192 7898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:08.983354 7898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:08.985348 7898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "e1fea5b70d0c4312a2e0082f2f1660ed"
format_stamp: "Formatted at 2026-09-28 06:45:08 on dist-test-slave-twl6"
server_key: "ad9db3db05aa6357d4871fb6e416d092"
server_key_iv: "a76d16eab4ec8fcfca9ecaccddea5e5e"
server_key_version: "encryptionkey@0"
I20260928 06:45:08.985841 7898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "e1fea5b70d0c4312a2e0082f2f1660ed"
format_stamp: "Formatted at 2026-09-28 06:45:08 on dist-test-slave-twl6"
server_key: "ad9db3db05aa6357d4871fb6e416d092"
server_key_iv: "a76d16eab4ec8fcfca9ecaccddea5e5e"
server_key_version: "encryptionkey@0"
I20260928 06:45:08.989358 7898 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 06:45:08.991627 7913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:08.992892 7898 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:08.992988 7898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "e1fea5b70d0c4312a2e0082f2f1660ed"
format_stamp: "Formatted at 2026-09-28 06:45:08 on dist-test-slave-twl6"
server_key: "ad9db3db05aa6357d4871fb6e416d092"
server_key_iv: "a76d16eab4ec8fcfca9ecaccddea5e5e"
server_key_version: "encryptionkey@0"
I20260928 06:45:08.993068 7898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:09.008625 7898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:09.012113 7898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:09.012328 7898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:09.019209 7898 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:46773
I20260928 06:45:09.019219 7965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:46773 every 8 connection(s)
I20260928 06:45:09.020249 7898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:45:09.022742 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 7898
I20260928 06:45:09.022833 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:45:09.023012 7966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:09.023133 7360 external_mini_cluster.cc:1428] Setting key 87b799f12f80497dfead359cce3cfab8
I20260928 06:45:09.030141 7966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1fea5b70d0c4312a2e0082f2f1660ed: Bootstrap starting.
Sep 28 06:45:09 dist-test-slave-twl6 krb5kdc[7885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:09.032662 7966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1fea5b70d0c4312a2e0082f2f1660ed: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:09.033533 7966 log.cc:824] T 00000000000000000000000000000000 P e1fea5b70d0c4312a2e0082f2f1660ed: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:09.035656 7966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1fea5b70d0c4312a2e0082f2f1660ed: No bootstrap required, opened a new log
I20260928 06:45:09.037573 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:09.024520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51028 (local address 127.7.48.62:46773)
0928 06:45:09.024930 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:09.024949 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:09.024993 (+ 44us) server_negotiation.cc:407] Connection header received
0928 06:45:09.025783 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:09.025808 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:09.026220 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:09.026584 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:09.027200 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.028689 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.029495 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.030170 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.030326 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:09.033051 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:09.033056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:09.033068 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:09.033100 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:09.034856 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:09.035335 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:09.035340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:09.035344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:09.035444 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:09.035756 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:09.035760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:09.035762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:09.036188 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:09.036335 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:09.036611 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 06:45:09.036888 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":102,"threads_started":1}
I20260928 06:45:09.039300 7966 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1fea5b70d0c4312a2e0082f2f1660ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1fea5b70d0c4312a2e0082f2f1660ed" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 46773 } }
I20260928 06:45:09.039527 7966 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1fea5b70d0c4312a2e0082f2f1660ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:09.039587 7966 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1fea5b70d0c4312a2e0082f2f1660ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1fea5b70d0c4312a2e0082f2f1660ed, State: Initialized, Role: FOLLOWER
I20260928 06:45:09.040032 7966 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1fea5b70d0c4312a2e0082f2f1660ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1fea5b70d0c4312a2e0082f2f1660ed" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 46773 } }
I20260928 06:45:09.040182 7966 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1fea5b70d0c4312a2e0082f2f1660ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:45:09.040243 7966 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1fea5b70d0c4312a2e0082f2f1660ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:45:09.040361 7966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1fea5b70d0c4312a2e0082f2f1660ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:09.041177 7966 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1fea5b70d0c4312a2e0082f2f1660ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1fea5b70d0c4312a2e0082f2f1660ed" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 46773 } }
I20260928 06:45:09.041492 7966 leader_election.cc:302] T 00000000000000000000000000000000 P e1fea5b70d0c4312a2e0082f2f1660ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1fea5b70d0c4312a2e0082f2f1660ed; no voters:
I20260928 06:45:09.041769 7966 leader_election.cc:288] T 00000000000000000000000000000000 P e1fea5b70d0c4312a2e0082f2f1660ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:45:09.041921 7971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1fea5b70d0c4312a2e0082f2f1660ed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:09.042152 7971 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1fea5b70d0c4312a2e0082f2f1660ed [term 1 LEADER]: Becoming Leader. State: Replica: e1fea5b70d0c4312a2e0082f2f1660ed, State: Running, Role: LEADER
I20260928 06:45:09.042466 7971 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1fea5b70d0c4312a2e0082f2f1660ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1fea5b70d0c4312a2e0082f2f1660ed" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 46773 } }
I20260928 06:45:09.042953 7966 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1fea5b70d0c4312a2e0082f2f1660ed [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:45:09.044071 7972 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1fea5b70d0c4312a2e0082f2f1660ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1fea5b70d0c4312a2e0082f2f1660ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1fea5b70d0c4312a2e0082f2f1660ed" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 46773 } } }
I20260928 06:45:09.044204 7972 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1fea5b70d0c4312a2e0082f2f1660ed [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:09.044498 7973 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1fea5b70d0c4312a2e0082f2f1660ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1fea5b70d0c4312a2e0082f2f1660ed. Latest consensus state: current_term: 1 leader_uuid: "e1fea5b70d0c4312a2e0082f2f1660ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1fea5b70d0c4312a2e0082f2f1660ed" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 46773 } } }
I20260928 06:45:09.044591 7973 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1fea5b70d0c4312a2e0082f2f1660ed [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:09.044741 7980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:45:09.047698 7980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:45:09.052853 7980 catalog_manager.cc:1347] Generated new cluster ID: 860f77979ab9413a8477318d8b2adaae
I20260928 06:45:09.052927 7980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:45:09.087659 7980 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:45:09.088886 7980 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:45:09.094307 7980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1fea5b70d0c4312a2e0082f2f1660ed: Generated new TSK 0
I20260928 06:45:09.094995 7980 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
I20260928 06:45:09.126823 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:46773
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:36505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:09.238339 7992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:09.238639 7992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:09.238721 7992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:09.242679 7992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:09.242770 7992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:09.242874 7992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:45:09.248263 7992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:36505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:46773
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:09.249332 7992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:09.250357 7992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:09.259668 8000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:09.259725 7998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:09.259725 7997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:09.260049 7992 server_base.cc:1034] running on GCE node
I20260928 06:45:09.260461 7992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:09.261070 7992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:09.262266 7992 hybrid_clock.cc:648] HybridClock initialized: now 1790577909262252 us; error 30 us; skew 500 ppm
Sep 28 06:45:09 dist-test-slave-twl6 krb5kdc[7885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577909, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:09.264979 7992 init.cc:377] Logged in from keytab as kudu/127.7.48.1@KRBTEST.COM (short username kudu)
I20260928 06:45:09.265852 7992 webserver.cc:460] Webserver started at http://127.7.48.1:37821/ using document root <none> and password file <none>
I20260928 06:45:09.266423 7992 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:09.266516 7992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:09.266762 7992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:09.268860 7992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "9bd273ea75f641ec868c0d0468f8f1a4"
format_stamp: "Formatted at 2026-09-28 06:45:09 on dist-test-slave-twl6"
server_key: "f918b35bf08e9f8c165116152ba99183"
server_key_iv: "a592f71a8be6d557d1c33e7133feb542"
server_key_version: "encryptionkey@0"
I20260928 06:45:09.269438 7992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "9bd273ea75f641ec868c0d0468f8f1a4"
format_stamp: "Formatted at 2026-09-28 06:45:09 on dist-test-slave-twl6"
server_key: "f918b35bf08e9f8c165116152ba99183"
server_key_iv: "a592f71a8be6d557d1c33e7133feb542"
server_key_version: "encryptionkey@0"
I20260928 06:45:09.273142 7992 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:45:09.275413 8007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:09.276655 7992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:09.276796 7992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "9bd273ea75f641ec868c0d0468f8f1a4"
format_stamp: "Formatted at 2026-09-28 06:45:09 on dist-test-slave-twl6"
server_key: "f918b35bf08e9f8c165116152ba99183"
server_key_iv: "a592f71a8be6d557d1c33e7133feb542"
server_key_version: "encryptionkey@0"
I20260928 06:45:09.276902 7992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:09.298206 7992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:09.300973 7992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:09.301203 7992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:09.301757 7992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:09.302556 7992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:09.302623 7992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:09.302692 7992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:09.302728 7992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:09.311199 7992 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:43405
I20260928 06:45:09.311223 8120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:43405 every 8 connection(s)
I20260928 06:45:09.312137 7992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:45:09.313086 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 7992
I20260928 06:45:09.313191 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
I20260928 06:45:09.313484 7360 external_mini_cluster.cc:1428] Setting key d3329971daa4b5a63c7b3c3f0183bba9
WARNING: no policy specified for kudu/127.7.48.2@KRBTEST.COM; defaulting to no policy
Sep 28 06:45:09 dist-test-slave-twl6 krb5kdc[7885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577909, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.2@KRBTEST.COM" created.
I20260928 06:45:09.327184 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:09.314114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:34859 (local address 127.7.48.62:46773)
0928 06:45:09.314272 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:09.314277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:09.315264 (+ 987us) server_negotiation.cc:407] Connection header received
0928 06:45:09.317175 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:09.317180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:09.317244 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:09.317344 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:09.318497 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.319418 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.320735 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.321176 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.321303 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:09.323625 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:09.323628 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:09.323630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:09.323656 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:09.325016 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:09.325542 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:09.325544 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:09.325546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:09.325662 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:09.326035 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:09.326037 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:09.326038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:09.326228 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:09.326306 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:09.326940 (+ 634us) server_negotiation.cc:299] Negotiation successful
0928 06:45:09.327045 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 06:45:09.328261 8123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:09.314547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:46773 (local address 127.7.48.1:34859)
0928 06:45:09.315088 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:09.315156 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:09.316966 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:09.317499 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:09.317506 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:09.317858 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:09.318342 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:09.318355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.319544 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.319547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:09.320609 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:09.320618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.321323 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.321325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:09.321464 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:09.322071 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:09.322088 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:09.323463 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:09.325165 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:09.325170 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:09.325180 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:09.325424 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:09.325781 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:09.325785 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:09.325788 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:09.325931 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:09.326337 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:09.326346 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:09.326683 (+ 337us) client_negotiation.cc:769] Sending connection context
0928 06:45:09.326907 (+ 224us) client_negotiation.cc:240] Negotiation successful
0928 06:45:09.327286 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":151,"threads_started":1}
I20260928 06:45:09.329566 8121 heartbeater.cc:347] Connected to a master server at 127.7.48.62:46773
I20260928 06:45:09.329811 8121 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:09.330327 8121 heartbeater.cc:511] Master 127.7.48.62:46773 requested a full tablet report, sending...
I20260928 06:45:09.331960 7930 ts_manager.cc:194] Registered new tserver with Master: 9bd273ea75f641ec868c0d0468f8f1a4 (127.7.48.1:43405)
I20260928 06:45:09.333189 7930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.1@KRBTEST.COM'} at 127.7.48.1:34859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
I20260928 06:45:09.343180 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:46773
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:36505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:09.454348 8126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:09.454640 8126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:09.454684 8126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:09.458654 8126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:09.458706 8126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:09.458765 8126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:45:09.463900 8126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:36505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:46773
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:09.464919 8126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:09.465914 8126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:09.475613 8132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:09.475667 8134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:09.475777 8126 server_base.cc:1034] running on GCE node
W20260928 06:45:09.475613 8131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:09.476352 8126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:09.476991 8126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:09.478169 8126 hybrid_clock.cc:648] HybridClock initialized: now 1790577909478155 us; error 26 us; skew 500 ppm
Sep 28 06:45:09 dist-test-slave-twl6 krb5kdc[7885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577909, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:09.481107 8126 init.cc:377] Logged in from keytab as kudu/127.7.48.2@KRBTEST.COM (short username kudu)
I20260928 06:45:09.481986 8126 webserver.cc:460] Webserver started at http://127.7.48.2:42463/ using document root <none> and password file <none>
I20260928 06:45:09.482566 8126 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:09.482666 8126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:09.482896 8126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:09.485144 8126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "1e461fd7ddf7483eaa65e1ee964cda04"
format_stamp: "Formatted at 2026-09-28 06:45:09 on dist-test-slave-twl6"
server_key: "016ada87ad857a60a3fa6c5ef0587562"
server_key_iv: "fa55e110bb837a358a827c7f9d69302b"
server_key_version: "encryptionkey@0"
I20260928 06:45:09.485726 8126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "1e461fd7ddf7483eaa65e1ee964cda04"
format_stamp: "Formatted at 2026-09-28 06:45:09 on dist-test-slave-twl6"
server_key: "016ada87ad857a60a3fa6c5ef0587562"
server_key_iv: "fa55e110bb837a358a827c7f9d69302b"
server_key_version: "encryptionkey@0"
I20260928 06:45:09.489447 8126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 06:45:09.491930 8141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:09.493155 8126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:45:09.493283 8126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "1e461fd7ddf7483eaa65e1ee964cda04"
format_stamp: "Formatted at 2026-09-28 06:45:09 on dist-test-slave-twl6"
server_key: "016ada87ad857a60a3fa6c5ef0587562"
server_key_iv: "fa55e110bb837a358a827c7f9d69302b"
server_key_version: "encryptionkey@0"
I20260928 06:45:09.493392 8126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:09.523449 8126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:09.526778 8126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:09.526986 8126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:09.527591 8126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:09.528442 8126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:09.528513 8126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:09.528584 8126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:09.528617 8126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:09.537492 8126 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:36593
I20260928 06:45:09.537523 8254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:36593 every 8 connection(s)
I20260928 06:45:09.538436 8126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:45:09.540081 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 8126
I20260928 06:45:09.540199 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:45:09.540448 7360 external_mini_cluster.cc:1428] Setting key 2b40f0ad87af504a89d04674da725f48
Sep 28 06:45:09 dist-test-slave-twl6 krb5kdc[7885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577909, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.48.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.3@KRBTEST.COM" created.
I20260928 06:45:09.552984 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:09.540353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:57307 (local address 127.7.48.62:46773)
0928 06:45:09.540492 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:09.540496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:09.541373 (+ 877us) server_negotiation.cc:407] Connection header received
0928 06:45:09.542959 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:09.542965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:09.543025 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:09.543116 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:09.544477 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.545250 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.546344 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.546560 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.546672 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:09.549146 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:09.549149 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:09.549151 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:09.549173 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:09.550622 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:09.551181 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:09.551186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:09.551188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:09.551267 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:09.551693 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:09.551697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:09.551698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:09.551919 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:09.552036 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:09.552696 (+ 660us) server_negotiation.cc:299] Negotiation successful
0928 06:45:09.552816 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 06:45:09.553798 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:09.540733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:46773 (local address 127.7.48.2:57307)
0928 06:45:09.541212 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:09.541269 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:09.542718 (+ 1449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:09.543279 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:09.543286 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:09.543747 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:09.544324 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:09.544338 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.545370 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.545373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:09.546221 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:09.546229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.546708 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.546712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:09.546884 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:09.547483 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:09.547498 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:09.548973 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:09.550764 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:09.550772 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:09.550788 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:09.551054 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:09.551360 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:09.551364 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:09.551366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:09.551556 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:09.552069 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:09.552077 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:09.552404 (+ 327us) client_negotiation.cc:769] Sending connection context
0928 06:45:09.552652 (+ 248us) client_negotiation.cc:240] Negotiation successful
0928 06:45:09.552957 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":121,"threads_started":1}
I20260928 06:45:09.554960 8255 heartbeater.cc:347] Connected to a master server at 127.7.48.62:46773
I20260928 06:45:09.555225 8255 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:09.555707 8255 heartbeater.cc:511] Master 127.7.48.62:46773 requested a full tablet report, sending...
I20260928 06:45:09.556828 7930 ts_manager.cc:194] Registered new tserver with Master: 1e461fd7ddf7483eaa65e1ee964cda04 (127.7.48.2:36593)
I20260928 06:45:09.557389 7930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.2@KRBTEST.COM'} at 127.7.48.2:57307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
I20260928 06:45:09.567194 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:46773
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:36505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:09.678100 8260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:09.678388 8260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:09.678428 8260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:09.682473 8260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:09.682533 8260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:09.682590 8260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:45:09.687804 8260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:36505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:46773
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:09.688835 8260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:09.689795 8260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:09.699342 8265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:09.699419 8260 server_base.cc:1034] running on GCE node
W20260928 06:45:09.699307 8268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:09.699314 8266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:09.700040 8260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:09.700660 8260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:09.701898 8260 hybrid_clock.cc:648] HybridClock initialized: now 1790577909701870 us; error 32 us; skew 500 ppm
Sep 28 06:45:09 dist-test-slave-twl6 krb5kdc[7885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577909, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:09.704610 8260 init.cc:377] Logged in from keytab as kudu/127.7.48.3@KRBTEST.COM (short username kudu)
I20260928 06:45:09.705443 8260 webserver.cc:460] Webserver started at http://127.7.48.3:34063/ using document root <none> and password file <none>
I20260928 06:45:09.706008 8260 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:09.706100 8260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:09.706333 8260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:09.708412 8260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "b969a6f54ca8408498fa76edb18396a9"
format_stamp: "Formatted at 2026-09-28 06:45:09 on dist-test-slave-twl6"
server_key: "7f8c42fdbfaf68e0a57d4d3ccb93364c"
server_key_iv: "59a11cb2165308674b90bf3611d430f4"
server_key_version: "encryptionkey@0"
I20260928 06:45:09.708981 8260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "b969a6f54ca8408498fa76edb18396a9"
format_stamp: "Formatted at 2026-09-28 06:45:09 on dist-test-slave-twl6"
server_key: "7f8c42fdbfaf68e0a57d4d3ccb93364c"
server_key_iv: "59a11cb2165308674b90bf3611d430f4"
server_key_version: "encryptionkey@0"
I20260928 06:45:09.712661 8260 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 06:45:09.715008 8275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:09.716256 8260 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:45:09.716384 8260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "b969a6f54ca8408498fa76edb18396a9"
format_stamp: "Formatted at 2026-09-28 06:45:09 on dist-test-slave-twl6"
server_key: "7f8c42fdbfaf68e0a57d4d3ccb93364c"
server_key_iv: "59a11cb2165308674b90bf3611d430f4"
server_key_version: "encryptionkey@0"
I20260928 06:45:09.716493 8260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:09.735683 8260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:09.738986 8260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:09.739204 8260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:09.739794 8260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:09.740694 8260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:09.740764 8260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:09.740837 8260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:09.740880 8260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:09.750352 8260 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:42455
I20260928 06:45:09.750375 8388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:42455 every 8 connection(s)
I20260928 06:45:09.751293 8260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:45:09.753298 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 8260
I20260928 06:45:09.753418 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:45:09.753741 7360 external_mini_cluster.cc:1428] Setting key 55a668d7958542ca8f576716e1b91c66
Sep 28 06:45:09 dist-test-slave-twl6 krb5kdc[7885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577909, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:09.765223 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:09.753257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:54525 (local address 127.7.48.62:46773)
0928 06:45:09.753408 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:09.753413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:09.754483 (+ 1070us) server_negotiation.cc:407] Connection header received
0928 06:45:09.755909 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:09.755912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:09.755970 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:09.756050 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:09.757244 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.757785 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.758994 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.759225 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.759336 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:09.761804 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:09.761808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:09.761813 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:09.761843 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:09.763068 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:09.763596 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:09.763600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:09.763602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:09.763690 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:09.764092 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:09.764097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:09.764102 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:09.764299 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:09.764410 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:09.764923 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 06:45:09.765054 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 06:45:09.766094 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:09.753825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:46773 (local address 127.7.48.3:54525)
0928 06:45:09.754345 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:09.754398 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:09.755654 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:09.756230 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:09.756238 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:09.756591 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:09.757084 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:09.757106 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.757927 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.757931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:09.758863 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:09.758872 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.759375 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.759380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:09.759562 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:09.760180 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:09.760195 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:09.761635 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:09.763185 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:09.763191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:09.763206 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:09.763480 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:09.763792 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:09.763795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:09.763797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:09.763997 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:09.764408 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:09.764415 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:09.764681 (+ 266us) client_negotiation.cc:769] Sending connection context
0928 06:45:09.764878 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 06:45:09.765225 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":133,"threads_started":1}
I20260928 06:45:09.767182 8389 heartbeater.cc:347] Connected to a master server at 127.7.48.62:46773
I20260928 06:45:09.767486 8389 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:09.768146 8389 heartbeater.cc:511] Master 127.7.48.62:46773 requested a full tablet report, sending...
I20260928 06:45:09.769208 7930 ts_manager.cc:194] Registered new tserver with Master: b969a6f54ca8408498fa76edb18396a9 (127.7.48.3:42455)
I20260928 06:45:09.769740 7930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.3@KRBTEST.COM'} at 127.7.48.3:54525
I20260928 06:45:09.779312 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:45:09.790759 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:09.783056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51030 (local address 127.7.48.62:46773)
0928 06:45:09.783217 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:09.783223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:09.783394 (+ 171us) server_negotiation.cc:407] Connection header received
0928 06:45:09.783561 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:09.783566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:09.783631 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:09.783717 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:09.784552 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.785067 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.785892 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.786111 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.786220 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:09.787681 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:09.787691 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:09.787694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:09.787719 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:09.789011 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:09.789444 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:09.789453 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:09.789454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:09.789530 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:09.789823 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:09.789827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:09.789829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:09.790058 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:09.790206 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:09.790449 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 06:45:09.790579 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:45:09.795583 7930 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:45:09.799015 7930 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:45:09.813052 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:09.807674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37548 (local address 127.7.48.2:36593)
0928 06:45:09.808003 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:09.808015 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:09.808553 (+ 538us) server_negotiation.cc:407] Connection header received
0928 06:45:09.808655 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:09.808660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:09.808825 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:09.808990 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:09.809384 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.810435 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.811763 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.812364 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.812491 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:09.812642 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:09.812806 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 06:45:09.812897 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":106,"threads_started":1}
I20260928 06:45:09.813042 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:09.807293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:36593 (local address 127.0.0.1:37548)
0928 06:45:09.808345 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:09.808358 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:09.808453 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:09.809175 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:09.809177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:09.809192 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:09.809265 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:09.809269 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.810566 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.810570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:09.811625 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:09.811641 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.812488 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.812493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:09.812595 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:09.812641 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 06:45:09.812776 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 06:45:09.812889 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":993,"mutex_wait_us":7,"thread_start_us":93,"threads_started":1}
I20260928 06:45:09.813215 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:09.807217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:43405 (local address 127.0.0.1:45092)
0928 06:45:09.807611 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:09.807653 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:09.807767 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:09.808641 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:09.808647 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:09.808672 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:09.808813 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:09.808824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.810354 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.810358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:09.811701 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:09.811712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.812851 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.812855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:09.812945 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:09.812968 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 06:45:09.813015 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 06:45:09.813077 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":82,"threads_started":1}
I20260928 06:45:09.813844 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:09.807331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45092 (local address 127.7.48.1:43405)
0928 06:45:09.808099 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:09.808115 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:09.808136 (+ 21us) server_negotiation.cc:407] Connection header received
0928 06:45:09.808214 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:09.808220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:09.808426 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:09.808978 (+ 552us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:09.809131 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.810225 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.811841 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.812721 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.813385 (+ 664us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:09.813549 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:09.813605 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 06:45:09.813696 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":76,"threads_started":1}
I20260928 06:45:09.815157 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:09.808328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:42455 (local address 127.0.0.1:39610)
0928 06:45:09.809431 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:09.809440 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:09.809539 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:09.809922 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:09.809925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:09.809940 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:09.810028 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:09.810034 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.811356 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.811359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:09.812469 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:09.812481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.814800 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.814804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:09.814885 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:09.814907 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 06:45:09.814983 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:45:09.815045 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058,"thread_start_us":94,"threads_started":1}
I20260928 06:45:09.815250 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:09.808752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39610 (local address 127.7.48.3:42455)
0928 06:45:09.809057 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:09.809068 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:09.809523 (+ 455us) server_negotiation.cc:407] Connection header received
0928 06:45:09.809653 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:09.809657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:09.809793 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:09.809927 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:09.810134 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.811206 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.812623 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:09.813939 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:09.814055 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:09.814286 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:09.815030 (+ 744us) server_negotiation.cc:299] Negotiation successful
0928 06:45:09.815114 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":97,"threads_started":1}
I20260928 06:45:09.816282 8189 tablet_service.cc:1467] Processing CreateTablet for tablet 86c526c0c8584928b76a7460695757ac (DEFAULT_TABLE table=test-table [id=07d8df2d765e4a809c19d5a721395d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:09.816655 8055 tablet_service.cc:1467] Processing CreateTablet for tablet 86c526c0c8584928b76a7460695757ac (DEFAULT_TABLE table=test-table [id=07d8df2d765e4a809c19d5a721395d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:09.817296 8189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86c526c0c8584928b76a7460695757ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:09.817687 8055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86c526c0c8584928b76a7460695757ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:09.817668 8323 tablet_service.cc:1467] Processing CreateTablet for tablet 86c526c0c8584928b76a7460695757ac (DEFAULT_TABLE table=test-table [id=07d8df2d765e4a809c19d5a721395d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:09.818607 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86c526c0c8584928b76a7460695757ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:09.823949 8406 tablet_bootstrap.cc:492] T 86c526c0c8584928b76a7460695757ac P 1e461fd7ddf7483eaa65e1ee964cda04: Bootstrap starting.
I20260928 06:45:09.824579 8407 tablet_bootstrap.cc:492] T 86c526c0c8584928b76a7460695757ac P 9bd273ea75f641ec868c0d0468f8f1a4: Bootstrap starting.
I20260928 06:45:09.825753 8408 tablet_bootstrap.cc:492] T 86c526c0c8584928b76a7460695757ac P b969a6f54ca8408498fa76edb18396a9: Bootstrap starting.
I20260928 06:45:09.826669 8407 tablet_bootstrap.cc:654] T 86c526c0c8584928b76a7460695757ac P 9bd273ea75f641ec868c0d0468f8f1a4: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:09.826932 8406 tablet_bootstrap.cc:654] T 86c526c0c8584928b76a7460695757ac P 1e461fd7ddf7483eaa65e1ee964cda04: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:09.827550 8408 tablet_bootstrap.cc:654] T 86c526c0c8584928b76a7460695757ac P b969a6f54ca8408498fa76edb18396a9: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:09.827838 8407 log.cc:824] T 86c526c0c8584928b76a7460695757ac P 9bd273ea75f641ec868c0d0468f8f1a4: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:09.827862 8406 log.cc:824] T 86c526c0c8584928b76a7460695757ac P 1e461fd7ddf7483eaa65e1ee964cda04: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:09.829057 8408 log.cc:824] T 86c526c0c8584928b76a7460695757ac P b969a6f54ca8408498fa76edb18396a9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:09.829916 8407 tablet_bootstrap.cc:492] T 86c526c0c8584928b76a7460695757ac P 9bd273ea75f641ec868c0d0468f8f1a4: No bootstrap required, opened a new log
I20260928 06:45:09.830090 8406 tablet_bootstrap.cc:492] T 86c526c0c8584928b76a7460695757ac P 1e461fd7ddf7483eaa65e1ee964cda04: No bootstrap required, opened a new log
I20260928 06:45:09.830121 8407 ts_tablet_manager.cc:1397] T 86c526c0c8584928b76a7460695757ac P 9bd273ea75f641ec868c0d0468f8f1a4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 06:45:09.830273 8406 ts_tablet_manager.cc:1397] T 86c526c0c8584928b76a7460695757ac P 1e461fd7ddf7483eaa65e1ee964cda04: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260928 06:45:09.830905 8408 tablet_bootstrap.cc:492] T 86c526c0c8584928b76a7460695757ac P b969a6f54ca8408498fa76edb18396a9: No bootstrap required, opened a new log
I20260928 06:45:09.831058 8408 ts_tablet_manager.cc:1397] T 86c526c0c8584928b76a7460695757ac P b969a6f54ca8408498fa76edb18396a9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 06:45:09.833101 8407 raft_consensus.cc:348] T 86c526c0c8584928b76a7460695757ac P 9bd273ea75f641ec868c0d0468f8f1a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd273ea75f641ec868c0d0468f8f1a4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 43405 } } peers { permanent_uuid: "1e461fd7ddf7483eaa65e1ee964cda04" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36593 } } peers { permanent_uuid: "b969a6f54ca8408498fa76edb18396a9" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 42455 } }
I20260928 06:45:09.833321 8407 raft_consensus.cc:374] T 86c526c0c8584928b76a7460695757ac P 9bd273ea75f641ec868c0d0468f8f1a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:09.833381 8407 raft_consensus.cc:729] T 86c526c0c8584928b76a7460695757ac P 9bd273ea75f641ec868c0d0468f8f1a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bd273ea75f641ec868c0d0468f8f1a4, State: Initialized, Role: FOLLOWER
I20260928 06:45:09.833863 8407 consensus_queue.cc:260] T 86c526c0c8584928b76a7460695757ac P 9bd273ea75f641ec868c0d0468f8f1a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd273ea75f641ec868c0d0468f8f1a4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 43405 } } peers { permanent_uuid: "1e461fd7ddf7483eaa65e1ee964cda04" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36593 } } peers { permanent_uuid: "b969a6f54ca8408498fa76edb18396a9" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 42455 } }
I20260928 06:45:09.834098 8406 raft_consensus.cc:348] T 86c526c0c8584928b76a7460695757ac P 1e461fd7ddf7483eaa65e1ee964cda04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd273ea75f641ec868c0d0468f8f1a4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 43405 } } peers { permanent_uuid: "1e461fd7ddf7483eaa65e1ee964cda04" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36593 } } peers { permanent_uuid: "b969a6f54ca8408498fa76edb18396a9" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 42455 } }
I20260928 06:45:09.834352 8406 raft_consensus.cc:374] T 86c526c0c8584928b76a7460695757ac P 1e461fd7ddf7483eaa65e1ee964cda04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:09.834407 8406 raft_consensus.cc:729] T 86c526c0c8584928b76a7460695757ac P 1e461fd7ddf7483eaa65e1ee964cda04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e461fd7ddf7483eaa65e1ee964cda04, State: Initialized, Role: FOLLOWER
I20260928 06:45:09.834424 8121 heartbeater.cc:503] Master 127.7.48.62:46773 was elected leader, sending a full tablet report...
I20260928 06:45:09.834802 8407 ts_tablet_manager.cc:1428] T 86c526c0c8584928b76a7460695757ac P 9bd273ea75f641ec868c0d0468f8f1a4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 06:45:09.834834 8408 raft_consensus.cc:348] T 86c526c0c8584928b76a7460695757ac P b969a6f54ca8408498fa76edb18396a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd273ea75f641ec868c0d0468f8f1a4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 43405 } } peers { permanent_uuid: "1e461fd7ddf7483eaa65e1ee964cda04" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36593 } } peers { permanent_uuid: "b969a6f54ca8408498fa76edb18396a9" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 42455 } }
I20260928 06:45:09.835081 8408 raft_consensus.cc:374] T 86c526c0c8584928b76a7460695757ac P b969a6f54ca8408498fa76edb18396a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:09.835165 8408 raft_consensus.cc:729] T 86c526c0c8584928b76a7460695757ac P b969a6f54ca8408498fa76edb18396a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b969a6f54ca8408498fa76edb18396a9, State: Initialized, Role: FOLLOWER
I20260928 06:45:09.835223 8406 consensus_queue.cc:260] T 86c526c0c8584928b76a7460695757ac P 1e461fd7ddf7483eaa65e1ee964cda04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd273ea75f641ec868c0d0468f8f1a4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 43405 } } peers { permanent_uuid: "1e461fd7ddf7483eaa65e1ee964cda04" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36593 } } peers { permanent_uuid: "b969a6f54ca8408498fa76edb18396a9" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 42455 } }
I20260928 06:45:09.835573 8408 consensus_queue.cc:260] T 86c526c0c8584928b76a7460695757ac P b969a6f54ca8408498fa76edb18396a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd273ea75f641ec868c0d0468f8f1a4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 43405 } } peers { permanent_uuid: "1e461fd7ddf7483eaa65e1ee964cda04" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36593 } } peers { permanent_uuid: "b969a6f54ca8408498fa76edb18396a9" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 42455 } }
I20260928 06:45:09.836184 8255 heartbeater.cc:503] Master 127.7.48.62:46773 was elected leader, sending a full tablet report...
I20260928 06:45:09.836231 8406 ts_tablet_manager.cc:1428] T 86c526c0c8584928b76a7460695757ac P 1e461fd7ddf7483eaa65e1ee964cda04: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260928 06:45:09.836503 8408 ts_tablet_manager.cc:1428] T 86c526c0c8584928b76a7460695757ac P b969a6f54ca8408498fa76edb18396a9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 06:45:09.836778 8389 heartbeater.cc:503] Master 127.7.48.62:46773 was elected leader, sending a full tablet report...
W20260928 06:45:10.002374 8390 tablet.cc:2369] T 86c526c0c8584928b76a7460695757ac P b969a6f54ca8408498fa76edb18396a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:45:10.007493 8412 raft_consensus.cc:482] T 86c526c0c8584928b76a7460695757ac P 9bd273ea75f641ec868c0d0468f8f1a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:45:10.007675 8412 raft_consensus.cc:504] T 86c526c0c8584928b76a7460695757ac P 9bd273ea75f641ec868c0d0468f8f1a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd273ea75f641ec868c0d0468f8f1a4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 43405 } } peers { permanent_uuid: "1e461fd7ddf7483eaa65e1ee964cda04" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36593 } } peers { permanent_uuid: "b969a6f54ca8408498fa76edb18396a9" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 42455 } }
I20260928 06:45:10.008797 8412 leader_election.cc:288] T 86c526c0c8584928b76a7460695757ac P 9bd273ea75f641ec868c0d0468f8f1a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e461fd7ddf7483eaa65e1ee964cda04 (127.7.48.2:36593), b969a6f54ca8408498fa76edb18396a9 (127.7.48.3:42455)
I20260928 06:45:10.012995 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:10.009201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:36127 (local address 127.7.48.2:36593)
0928 06:45:10.009383 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:10.009390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:10.009486 (+ 96us) server_negotiation.cc:407] Connection header received
0928 06:45:10.009721 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:10.009727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:10.009790 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:10.009877 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:10.010241 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:10.010840 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:10.011985 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:10.012415 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:10.012493 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:10.012566 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:10.012795 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 06:45:10.012868 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 06:45:10.013003 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:10.009711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:42055 (local address 127.7.48.3:42455)
0928 06:45:10.009880 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:10.009886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:10.009966 (+ 80us) server_negotiation.cc:407] Connection header received
0928 06:45:10.010074 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:10.010080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:10.010123 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:10.010209 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:10.010479 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:10.010980 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:10.012048 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:10.012457 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:10.012580 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:10.012636 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:10.012838 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 06:45:10.012907 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 06:45:10.013027 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:10.009077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:36593 (local address 127.7.48.1:36127)
0928 06:45:10.009387 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:10.009401 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:10.009539 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:10.009912 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:10.009915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:10.009961 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:10.010108 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:10.010116 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:10.010963 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:10.010966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:10.011837 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:10.011846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:10.012551 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:10.012556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:10.012663 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:10.012687 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 06:45:10.012754 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:45:10.012842 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":2,"thread_start_us":107,"threads_started":1}
I20260928 06:45:10.013010 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:10.009543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:42455 (local address 127.7.48.1:42055)
0928 06:45:10.009881 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:10.009896 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:10.009987 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:10.010239 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:10.010241 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:10.010258 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:10.010371 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:10.010377 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:10.011084 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:10.011094 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:10.011877 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:10.011885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:10.012603 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:10.012607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:10.012696 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:10.012713 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 06:45:10.012806 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 06:45:10.012867 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":122,"threads_started":1}
I20260928 06:45:10.013751 8209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86c526c0c8584928b76a7460695757ac" candidate_uuid: "9bd273ea75f641ec868c0d0468f8f1a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e461fd7ddf7483eaa65e1ee964cda04" is_pre_election: true
I20260928 06:45:10.013871 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86c526c0c8584928b76a7460695757ac" candidate_uuid: "9bd273ea75f641ec868c0d0468f8f1a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b969a6f54ca8408498fa76edb18396a9" is_pre_election: true
I20260928 06:45:10.014106 8209 raft_consensus.cc:2454] T 86c526c0c8584928b76a7460695757ac P 1e461fd7ddf7483eaa65e1ee964cda04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bd273ea75f641ec868c0d0468f8f1a4 in term 0.
I20260928 06:45:10.014166 8343 raft_consensus.cc:2454] T 86c526c0c8584928b76a7460695757ac P b969a6f54ca8408498fa76edb18396a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bd273ea75f641ec868c0d0468f8f1a4 in term 0.
I20260928 06:45:10.014585 8009 leader_election.cc:302] T 86c526c0c8584928b76a7460695757ac P 9bd273ea75f641ec868c0d0468f8f1a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e461fd7ddf7483eaa65e1ee964cda04, 9bd273ea75f641ec868c0d0468f8f1a4; no voters:
I20260928 06:45:10.014869 8412 raft_consensus.cc:2781] T 86c526c0c8584928b76a7460695757ac P 9bd273ea75f641ec868c0d0468f8f1a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:45:10.014978 8412 raft_consensus.cc:482] T 86c526c0c8584928b76a7460695757ac P 9bd273ea75f641ec868c0d0468f8f1a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:45:10.015025 8412 raft_consensus.cc:3037] T 86c526c0c8584928b76a7460695757ac P 9bd273ea75f641ec868c0d0468f8f1a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:10.016072 8412 raft_consensus.cc:504] T 86c526c0c8584928b76a7460695757ac P 9bd273ea75f641ec868c0d0468f8f1a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd273ea75f641ec868c0d0468f8f1a4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 43405 } } peers { permanent_uuid: "1e461fd7ddf7483eaa65e1ee964cda04" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36593 } } peers { permanent_uuid: "b969a6f54ca8408498fa76edb18396a9" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 42455 } }
I20260928 06:45:10.016403 8412 leader_election.cc:288] T 86c526c0c8584928b76a7460695757ac P 9bd273ea75f641ec868c0d0468f8f1a4 [CANDIDATE]: Term 1 election: Requested vote from peers 1e461fd7ddf7483eaa65e1ee964cda04 (127.7.48.2:36593), b969a6f54ca8408498fa76edb18396a9 (127.7.48.3:42455)
I20260928 06:45:10.016741 8209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86c526c0c8584928b76a7460695757ac" candidate_uuid: "9bd273ea75f641ec868c0d0468f8f1a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e461fd7ddf7483eaa65e1ee964cda04"
I20260928 06:45:10.016815 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86c526c0c8584928b76a7460695757ac" candidate_uuid: "9bd273ea75f641ec868c0d0468f8f1a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b969a6f54ca8408498fa76edb18396a9"
I20260928 06:45:10.016886 8209 raft_consensus.cc:3037] T 86c526c0c8584928b76a7460695757ac P 1e461fd7ddf7483eaa65e1ee964cda04 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:10.016959 8343 raft_consensus.cc:3037] T 86c526c0c8584928b76a7460695757ac P b969a6f54ca8408498fa76edb18396a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:10.017922 8209 raft_consensus.cc:2454] T 86c526c0c8584928b76a7460695757ac P 1e461fd7ddf7483eaa65e1ee964cda04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bd273ea75f641ec868c0d0468f8f1a4 in term 1.
I20260928 06:45:10.017961 8343 raft_consensus.cc:2454] T 86c526c0c8584928b76a7460695757ac P b969a6f54ca8408498fa76edb18396a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bd273ea75f641ec868c0d0468f8f1a4 in term 1.
I20260928 06:45:10.018246 8009 leader_election.cc:302] T 86c526c0c8584928b76a7460695757ac P 9bd273ea75f641ec868c0d0468f8f1a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e461fd7ddf7483eaa65e1ee964cda04, 9bd273ea75f641ec868c0d0468f8f1a4; no voters:
I20260928 06:45:10.018456 8412 raft_consensus.cc:2781] T 86c526c0c8584928b76a7460695757ac P 9bd273ea75f641ec868c0d0468f8f1a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:10.018712 8412 raft_consensus.cc:686] T 86c526c0c8584928b76a7460695757ac P 9bd273ea75f641ec868c0d0468f8f1a4 [term 1 LEADER]: Becoming Leader. State: Replica: 9bd273ea75f641ec868c0d0468f8f1a4, State: Running, Role: LEADER
I20260928 06:45:10.019106 8412 consensus_queue.cc:237] T 86c526c0c8584928b76a7460695757ac P 9bd273ea75f641ec868c0d0468f8f1a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd273ea75f641ec868c0d0468f8f1a4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 43405 } } peers { permanent_uuid: "1e461fd7ddf7483eaa65e1ee964cda04" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36593 } } peers { permanent_uuid: "b969a6f54ca8408498fa76edb18396a9" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 42455 } }
I20260928 06:45:10.022459 7929 catalog_manager.cc:5495] T 86c526c0c8584928b76a7460695757ac P 9bd273ea75f641ec868c0d0468f8f1a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bd273ea75f641ec868c0d0468f8f1a4 (127.7.48.1). New cstate: current_term: 1 leader_uuid: "9bd273ea75f641ec868c0d0468f8f1a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd273ea75f641ec868c0d0468f8f1a4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 43405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e461fd7ddf7483eaa65e1ee964cda04" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b969a6f54ca8408498fa76edb18396a9" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 42455 } health_report { overall_health: UNKNOWN } } }
W20260928 06:45:10.039685 8256 tablet.cc:2369] T 86c526c0c8584928b76a7460695757ac P 1e461fd7ddf7483eaa65e1ee964cda04: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 06:45:10 dist-test-slave-twl6 krb5kdc[7885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.1@KRBTEST.COM
I20260928 06:45:10.048352 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:10.039783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45108 (local address 127.7.48.1:43405)
0928 06:45:10.039926 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:10.039930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:10.040088 (+ 158us) server_negotiation.cc:407] Connection header received
0928 06:45:10.040262 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:10.040264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:10.040312 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:10.040387 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:10.041037 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:10.041563 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:10.042421 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:10.042669 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:10.042783 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:10.045365 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:10.045374 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:10.045387 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:10.045408 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:10.046876 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:10.047277 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:10.047282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:10.047283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:10.047354 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:10.047607 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:10.047610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:10.047611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:10.047821 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:10.048022 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:10.048155 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 06:45:10.048243 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 06:45:10 dist-test-slave-twl6 krb5kdc[7885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577910, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260928 06:45:10.063730 8122 tablet.cc:2369] T 86c526c0c8584928b76a7460695757ac P 9bd273ea75f641ec868c0d0468f8f1a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 06:45:10 dist-test-slave-twl6 krb5kdc[7885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577910, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.48.1@KRBTEST.COM
I20260928 06:45:10.074293 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:10.065724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45124 (local address 127.7.48.1:43405)
0928 06:45:10.065846 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:10.065850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:10.066004 (+ 154us) server_negotiation.cc:407] Connection header received
0928 06:45:10.066182 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:10.066185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:10.066223 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:10.066306 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:10.066985 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:10.067479 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:10.068343 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:10.068566 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:10.068673 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:10.071379 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:10.071391 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:10.071393 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:10.071413 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:10.072661 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:10.073053 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:10.073056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:10.073058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:10.073139 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:10.073408 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:10.073411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:10.073413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:10.073597 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:10.073732 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:10.074014 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 06:45:10.074128 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 06:45:10 dist-test-slave-twl6 krb5kdc[7885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:45:10 dist-test-slave-twl6 krb5kdc[7885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.1@KRBTEST.COM
I20260928 06:45:10.098778 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:10.090548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45138 (local address 127.7.48.1:43405)
0928 06:45:10.090707 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:10.090711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:10.090865 (+ 154us) server_negotiation.cc:407] Connection header received
0928 06:45:10.091030 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:10.091032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:10.091071 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:10.091163 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:10.091814 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:10.092373 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:10.093242 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:10.093472 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:10.093577 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:10.095841 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:10.095844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:10.095864 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:10.095921 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:10.097264 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:10.097627 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:10.097629 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:10.097630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:10.097699 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:10.097962 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:10.097964 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:10.097965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:10.098113 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:10.098251 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:10.098462 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 06:45:10.098552 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 06:45:10.100178 8035 tablet_service.cc:2908] Rejecting scan request for tablet 86c526c0c8584928b76a7460695757ac: Uninitialized: clean time has not yet been initialized
I20260928 06:45:10.102054 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 7992
I20260928 06:45:10.108075 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 8126
I20260928 06:45:10.113422 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 8260
I20260928 06:45:10.118868 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 7898
2026-09-28T06:45:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2421 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:45:10 dist-test-slave-twl6 krb5kdc[8431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:45:10 dist-test-slave-twl6 krb5kdc[8431](info): set up 2 sockets
Sep 28 06:45:10 dist-test-slave-twl6 krb5kdc[8431](info): commencing operation
krb5kdc: starting...
W20260928 06:45:11.144249 7360 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:45:11 dist-test-slave-twl6 krb5kdc[8431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:45:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:45:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:45:11.260792 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:36049
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:38581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:36049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:11.368157 8444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:11.368384 8444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:11.368422 8444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:11.372390 8444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:45:11.372432 8444 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 06:45:11.372449 8444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:11.372500 8444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:11.372545 8444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:45:11.372577 8444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:45:11.377632 8444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:38581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:36049
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:36049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:11.378546 8444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:11.379474 8444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:11.387724 8449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:11.387768 8452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:11.387771 8444 server_base.cc:1034] running on GCE node
W20260928 06:45:11.388052 8450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:11.388581 8444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:11.389663 8444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:11.390847 8444 hybrid_clock.cc:648] HybridClock initialized: now 1790577911390830 us; error 30 us; skew 500 ppm
Sep 28 06:45:11 dist-test-slave-twl6 krb5kdc[8431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577911, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:11.398398 8444 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:45:11.399295 8444 webserver.cc:460] Webserver started at http://127.7.48.62:38961/ using document root <none> and password file <none>
I20260928 06:45:11.399855 8444 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:11.399995 8444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:11.400231 8444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:11.402309 8444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "a52b5269fe354aef9616d50180bd5505"
format_stamp: "Formatted at 2026-09-28 06:45:11 on dist-test-slave-twl6"
server_key: "a604db78865eb483885951fecbb82f73"
server_key_iv: "4eeb887a78779d4dd4c962d3a1d29e85"
server_key_version: "encryptionkey@0"
I20260928 06:45:11.402872 8444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "a52b5269fe354aef9616d50180bd5505"
format_stamp: "Formatted at 2026-09-28 06:45:11 on dist-test-slave-twl6"
server_key: "a604db78865eb483885951fecbb82f73"
server_key_iv: "4eeb887a78779d4dd4c962d3a1d29e85"
server_key_version: "encryptionkey@0"
I20260928 06:45:11.406525 8444 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 06:45:11.408936 8459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:11.410095 8444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:45:11.410238 8444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "a52b5269fe354aef9616d50180bd5505"
format_stamp: "Formatted at 2026-09-28 06:45:11 on dist-test-slave-twl6"
server_key: "a604db78865eb483885951fecbb82f73"
server_key_iv: "4eeb887a78779d4dd4c962d3a1d29e85"
server_key_version: "encryptionkey@0"
I20260928 06:45:11.410351 8444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:11.428644 8444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:11.431524 8444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:11.431748 8444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:11.439931 8444 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:36049
I20260928 06:45:11.439934 8521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:36049 every 8 connection(s)
I20260928 06:45:11.440944 8444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:45:11.443682 8522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:11.446622 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 8444
I20260928 06:45:11.446755 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:45:11.447022 7360 external_mini_cluster.cc:1428] Setting key 8c2ef152ac749ea9a2737bd4e1920559
I20260928 06:45:11.449476 8522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a52b5269fe354aef9616d50180bd5505: Bootstrap starting.
I20260928 06:45:11.452040 8522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a52b5269fe354aef9616d50180bd5505: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:11.452768 8522 log.cc:824] T 00000000000000000000000000000000 P a52b5269fe354aef9616d50180bd5505: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:11.454569 8522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a52b5269fe354aef9616d50180bd5505: No bootstrap required, opened a new log
Sep 28 06:45:11 dist-test-slave-twl6 krb5kdc[8431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:11.458673 8522 raft_consensus.cc:348] T 00000000000000000000000000000000 P a52b5269fe354aef9616d50180bd5505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52b5269fe354aef9616d50180bd5505" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36049 } }
I20260928 06:45:11.458957 8522 raft_consensus.cc:374] T 00000000000000000000000000000000 P a52b5269fe354aef9616d50180bd5505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:11.459043 8522 raft_consensus.cc:729] T 00000000000000000000000000000000 P a52b5269fe354aef9616d50180bd5505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a52b5269fe354aef9616d50180bd5505, State: Initialized, Role: FOLLOWER
I20260928 06:45:11.459529 8522 consensus_queue.cc:260] T 00000000000000000000000000000000 P a52b5269fe354aef9616d50180bd5505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52b5269fe354aef9616d50180bd5505" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36049 } }
I20260928 06:45:11.459713 8522 raft_consensus.cc:388] T 00000000000000000000000000000000 P a52b5269fe354aef9616d50180bd5505 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:45:11.459789 8522 raft_consensus.cc:482] T 00000000000000000000000000000000 P a52b5269fe354aef9616d50180bd5505 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:45:11.459916 8522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a52b5269fe354aef9616d50180bd5505 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:11.460742 8522 raft_consensus.cc:504] T 00000000000000000000000000000000 P a52b5269fe354aef9616d50180bd5505 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52b5269fe354aef9616d50180bd5505" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36049 } }
I20260928 06:45:11.460843 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:11.448371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35846 (local address 127.7.48.62:36049)
0928 06:45:11.448785 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:11.448800 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:11.448835 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:45:11.449427 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:11.449443 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:11.449819 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:11.450092 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:11.450994 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:11.452039 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:11.452885 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:11.453358 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:11.453548 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:11.456298 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:11.456303 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:11.456315 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:11.456339 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:11.457961 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:11.458427 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:11.458432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:11.458436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:11.458558 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:11.458841 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:11.458844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:11.458845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:11.459306 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:11.459463 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:11.459758 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 06:45:11.460117 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":129,"threads_started":1}
I20260928 06:45:11.461097 8522 leader_election.cc:302] T 00000000000000000000000000000000 P a52b5269fe354aef9616d50180bd5505 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a52b5269fe354aef9616d50180bd5505; no voters:
I20260928 06:45:11.461396 8522 leader_election.cc:288] T 00000000000000000000000000000000 P a52b5269fe354aef9616d50180bd5505 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:45:11.461498 8527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a52b5269fe354aef9616d50180bd5505 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:11.461717 8527 raft_consensus.cc:686] T 00000000000000000000000000000000 P a52b5269fe354aef9616d50180bd5505 [term 1 LEADER]: Becoming Leader. State: Replica: a52b5269fe354aef9616d50180bd5505, State: Running, Role: LEADER
I20260928 06:45:11.462008 8527 consensus_queue.cc:237] T 00000000000000000000000000000000 P a52b5269fe354aef9616d50180bd5505 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52b5269fe354aef9616d50180bd5505" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36049 } }
I20260928 06:45:11.462595 8522 sys_catalog.cc:564] T 00000000000000000000000000000000 P a52b5269fe354aef9616d50180bd5505 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:45:11.463150 8527 sys_catalog.cc:455] T 00000000000000000000000000000000 P a52b5269fe354aef9616d50180bd5505 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a52b5269fe354aef9616d50180bd5505. Latest consensus state: current_term: 1 leader_uuid: "a52b5269fe354aef9616d50180bd5505" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52b5269fe354aef9616d50180bd5505" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36049 } } }
I20260928 06:45:11.463254 8527 sys_catalog.cc:458] T 00000000000000000000000000000000 P a52b5269fe354aef9616d50180bd5505 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:11.463558 8528 sys_catalog.cc:455] T 00000000000000000000000000000000 P a52b5269fe354aef9616d50180bd5505 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a52b5269fe354aef9616d50180bd5505" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52b5269fe354aef9616d50180bd5505" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36049 } } }
I20260928 06:45:11.463661 8528 sys_catalog.cc:458] T 00000000000000000000000000000000 P a52b5269fe354aef9616d50180bd5505 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:11.463685 8532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:45:11.466745 8532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:45:11.472201 8532 catalog_manager.cc:1347] Generated new cluster ID: cac5479b4f7f4702a78a39a1cb9bbcea
I20260928 06:45:11.472265 8532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:45:11.482110 8532 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:45:11.483315 8532 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:45:11.497218 8532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a52b5269fe354aef9616d50180bd5505: Generated new TSK 0
I20260928 06:45:11.498066 8532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
I20260928 06:45:11.532156 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:36049
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:38581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:11.641284 8548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:11.641569 8548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:11.641630 8548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:11.645610 8548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:11.645668 8548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:11.645704 8548 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:45:11.645737 8548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:45:11.650909 8548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:38581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.48.62:36049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:11.651969 8548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:11.652969 8548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:11.662408 8554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:11.662420 8553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:11.662420 8556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:11.662482 8548 server_base.cc:1034] running on GCE node
I20260928 06:45:11.663050 8548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:11.663733 8548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:11.664913 8548 hybrid_clock.cc:648] HybridClock initialized: now 1790577911664905 us; error 29 us; skew 500 ppm
Sep 28 06:45:11 dist-test-slave-twl6 krb5kdc[8431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577911, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:11.667454 8548 init.cc:377] Logged in from keytab as kudu/127.7.48.1@KRBTEST.COM (short username kudu)
I20260928 06:45:11.668371 8548 webserver.cc:460] Webserver started at http://127.7.48.1:36235/ using document root <none> and password file <none>
I20260928 06:45:11.668913 8548 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:11.668999 8548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:11.669209 8548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:11.671162 8548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "4526e1b23db04332929ca7d58baac1ba"
format_stamp: "Formatted at 2026-09-28 06:45:11 on dist-test-slave-twl6"
server_key: "efc6b9e5f7cf8cac7846890ef541b7bd"
server_key_iv: "2f12d31ba7f89a367b2375eafaa85cf0"
server_key_version: "encryptionkey@0"
I20260928 06:45:11.671741 8548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "4526e1b23db04332929ca7d58baac1ba"
format_stamp: "Formatted at 2026-09-28 06:45:11 on dist-test-slave-twl6"
server_key: "efc6b9e5f7cf8cac7846890ef541b7bd"
server_key_iv: "2f12d31ba7f89a367b2375eafaa85cf0"
server_key_version: "encryptionkey@0"
I20260928 06:45:11.675352 8548 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 06:45:11.677748 8563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:11.678851 8548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:11.678978 8548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "4526e1b23db04332929ca7d58baac1ba"
format_stamp: "Formatted at 2026-09-28 06:45:11 on dist-test-slave-twl6"
server_key: "efc6b9e5f7cf8cac7846890ef541b7bd"
server_key_iv: "2f12d31ba7f89a367b2375eafaa85cf0"
server_key_version: "encryptionkey@0"
I20260928 06:45:11.679090 8548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:11.695832 8548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:11.698565 8548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:11.698770 8548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:11.700246 8548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:11.700328 8548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:11.700408 8548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:11.700449 8548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:45:11 dist-test-slave-twl6 krb5kdc[8431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577911, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:11.711921 8548 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:42719
I20260928 06:45:11.711998 8678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:42719 every 8 connection(s)
I20260928 06:45:11.712983 8548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:45:11.714172 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:11.700945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:38417 (local address 127.7.48.62:36049)
0928 06:45:11.701070 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:11.701074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:11.702660 (+ 1586us) server_negotiation.cc:407] Connection header received
0928 06:45:11.703912 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:11.703916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:11.703972 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:11.704025 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:11.705488 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:11.706082 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:11.707256 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:11.707641 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:11.707775 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:11.710413 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:11.710417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:11.710420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:11.710452 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:11.712106 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:11.712566 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:11.712573 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:11.712577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:11.712679 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:11.713013 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:11.713019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:11.713022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:11.713235 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:11.713346 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:11.713907 (+ 561us) server_negotiation.cc:299] Negotiation successful
0928 06:45:11.714032 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:45:11.714998 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:11.702005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:36049 (local address 127.7.48.1:38417)
0928 06:45:11.702490 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:11.702545 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:11.703643 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:11.704394 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:11.704402 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:11.704863 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:11.705342 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:11.705355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:11.706216 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:11.706219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:11.707129 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:11.707138 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:11.707834 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:11.707839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:11.708065 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:11.708697 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:11.708719 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:11.710260 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:11.712205 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:11.712209 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:11.712220 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:11.712448 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:11.712771 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:11.712773 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:11.712775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:11.712915 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:11.713360 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:11.713367 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:11.713657 (+ 290us) client_negotiation.cc:769] Sending connection context
0928 06:45:11.713858 (+ 201us) client_negotiation.cc:240] Negotiation successful
0928 06:45:11.714140 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":124,"threads_started":1}
I20260928 06:45:11.716172 8679 heartbeater.cc:347] Connected to a master server at 127.7.48.62:36049
I20260928 06:45:11.716377 8679 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:11.716835 8679 heartbeater.cc:511] Master 127.7.48.62:36049 requested a full tablet report, sending...
I20260928 06:45:11.717931 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 8548
I20260928 06:45:11.718050 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
I20260928 06:45:11.718348 7360 external_mini_cluster.cc:1428] Setting key c5ec93cfdde5a686526ca324df6b9d97
I20260928 06:45:11.718384 8476 ts_manager.cc:194] Registered new tserver with Master: 4526e1b23db04332929ca7d58baac1ba (127.7.48.1:42719)
I20260928 06:45:11.719784 8476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.1@KRBTEST.COM'} at 127.7.48.1:38417
I20260928 06:45:11.726462 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:11.718542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:36049 (local address 127.7.48.1:36317)
0928 06:45:11.718866 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:11.718881 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:11.718981 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:11.719248 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:11.719250 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:11.719396 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:11.719482 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:11.719486 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:11.720448 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:11.720451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:11.721306 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:11.721317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:11.722078 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:11.722085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:11.722224 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:11.722724 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:11.722745 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:11.723117 (+ 372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:11.725066 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:11.725068 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:11.725070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:11.725263 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:11.725557 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:11.725564 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:11.725566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:11.725683 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:11.726092 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:11.726094 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:11.726164 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 06:45:11.726240 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:45:11.726333 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":121,"threads_started":1}
I20260928 06:45:11.726689 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:11.718908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:36317 (local address 127.7.48.62:36049)
0928 06:45:11.719043 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:11.719047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:11.719060 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:45:11.719093 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:11.719095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:11.719147 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:11.719227 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:11.719579 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:11.720290 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:11.721447 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:11.721817 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:11.721911 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:11.723264 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:11.723271 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:11.723275 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:11.723306 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:11.724949 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:11.725364 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:11.725370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:11.725373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:11.725462 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:11.725778 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:11.725781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:11.725783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:11.725993 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:11.726114 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:11.726331 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 06:45:11.726459 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
WARNING: no policy specified for kudu/127.7.48.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
I20260928 06:45:11.746685 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:36049
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:38581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:11.855918 8688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:11.856155 8688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:11.856196 8688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:11.860211 8688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:11.860261 8688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:11.860325 8688 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:45:11.860409 8688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:45:11.865692 8688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:38581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.48.62:36049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:11.866701 8688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:11.867726 8688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:11.877210 8693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:11.877312 8696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:11.877413 8688 server_base.cc:1034] running on GCE node
W20260928 06:45:11.877534 8694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:11.877941 8688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:11.878569 8688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:11.879756 8688 hybrid_clock.cc:648] HybridClock initialized: now 1790577911879749 us; error 31 us; skew 500 ppm
Sep 28 06:45:11 dist-test-slave-twl6 krb5kdc[8431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577911, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:11.882592 8688 init.cc:377] Logged in from keytab as kudu/127.7.48.2@KRBTEST.COM (short username kudu)
I20260928 06:45:11.883502 8688 webserver.cc:460] Webserver started at http://127.7.48.2:44587/ using document root <none> and password file <none>
I20260928 06:45:11.884115 8688 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:11.884230 8688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:11.884461 8688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:11.886494 8688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "22879aa1a4e3406e87590c3bf4bc96d8"
format_stamp: "Formatted at 2026-09-28 06:45:11 on dist-test-slave-twl6"
server_key: "af2aec944170cbc46dea35517545eb93"
server_key_iv: "405a47c82cbba2973e3647e8e4353993"
server_key_version: "encryptionkey@0"
I20260928 06:45:11.887053 8688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "22879aa1a4e3406e87590c3bf4bc96d8"
format_stamp: "Formatted at 2026-09-28 06:45:11 on dist-test-slave-twl6"
server_key: "af2aec944170cbc46dea35517545eb93"
server_key_iv: "405a47c82cbba2973e3647e8e4353993"
server_key_version: "encryptionkey@0"
I20260928 06:45:11.890697 8688 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 06:45:11.893158 8703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:11.894304 8688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:45:11.894433 8688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "22879aa1a4e3406e87590c3bf4bc96d8"
format_stamp: "Formatted at 2026-09-28 06:45:11 on dist-test-slave-twl6"
server_key: "af2aec944170cbc46dea35517545eb93"
server_key_iv: "405a47c82cbba2973e3647e8e4353993"
server_key_version: "encryptionkey@0"
I20260928 06:45:11.894536 8688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:11.926177 8688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:11.929644 8688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:11.929847 8688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:11.931296 8688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:11.931398 8688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:11.931488 8688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:11.931540 8688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:45:11 dist-test-slave-twl6 krb5kdc[8431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577911, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:11.942906 8688 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:45041
I20260928 06:45:11.943181 8818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:45041 every 8 connection(s)
I20260928 06:45:11.943816 8688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:45:11.945858 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:11.932230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:55413 (local address 127.7.48.62:36049)
0928 06:45:11.932363 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:11.932370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:11.933406 (+ 1036us) server_negotiation.cc:407] Connection header received
0928 06:45:11.934550 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:11.934554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:11.934622 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:11.934692 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:11.936478 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:11.937337 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:11.938834 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:11.939192 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:11.939302 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:11.941954 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:11.941957 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:11.941959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:11.941983 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:11.943675 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:11.944295 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:11.944299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:11.944301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:11.944391 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:11.944841 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:11.944845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:11.944846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:11.945078 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:11.945172 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:11.945640 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 06:45:11.945736 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:45:11.946624 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:11.932793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:36049 (local address 127.7.48.2:55413)
0928 06:45:11.933265 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:11.933306 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:11.934291 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:11.935130 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:11.935141 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:11.935644 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:11.936290 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:11.936310 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:11.937477 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:11.937482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:11.938726 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:11.938737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:11.939351 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:11.939355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:11.939552 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:11.940152 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:11.940167 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:11.941724 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:11.943803 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:11.943810 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:11.943823 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:11.944184 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:11.944517 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:11.944522 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:11.944526 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:11.944714 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:11.945182 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:11.945186 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:11.945400 (+ 214us) client_negotiation.cc:769] Sending connection context
0928 06:45:11.945572 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 06:45:11.945809 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":137,"threads_started":1}
I20260928 06:45:11.947805 8819 heartbeater.cc:347] Connected to a master server at 127.7.48.62:36049
I20260928 06:45:11.948004 8819 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:11.948523 8819 heartbeater.cc:511] Master 127.7.48.62:36049 requested a full tablet report, sending...
I20260928 06:45:11.949563 8476 ts_manager.cc:194] Registered new tserver with Master: 22879aa1a4e3406e87590c3bf4bc96d8 (127.7.48.2:45041)
I20260928 06:45:11.950322 8476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.2@KRBTEST.COM'} at 127.7.48.2:55413
I20260928 06:45:11.952800 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 8688
I20260928 06:45:11.952937 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:45:11.953243 7360 external_mini_cluster.cc:1428] Setting key 8500c6be6b5ae1ee47c01f7b5f6fc1b9
I20260928 06:45:11.957638 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:11.950042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:35905 (local address 127.7.48.62:36049)
0928 06:45:11.950153 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:11.950156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:11.950788 (+ 632us) server_negotiation.cc:407] Connection header received
0928 06:45:11.950908 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:11.950911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:11.950966 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:11.951049 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:11.951445 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:11.952190 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:11.953000 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:11.953315 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:11.953442 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:11.954544 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:11.954548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:11.954557 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:11.954578 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:11.956020 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:11.956423 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:11.956427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:11.956428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:11.956508 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:11.956792 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:11.956797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:11.956799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:11.957010 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:11.957115 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:11.957371 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 06:45:11.957494 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:45:11.957751 8825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:11.949976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:36049 (local address 127.7.48.2:35905)
0928 06:45:11.950700 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:11.950715 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:11.950811 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:11.951072 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:11.951074 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:11.951239 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:11.951332 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:11.951337 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:11.952310 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:11.952312 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:11.952866 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:11.952874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:11.953437 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:11.953440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:11.953507 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:11.954030 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:11.954049 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:11.954419 (+ 370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:11.956124 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:11.956128 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:11.956131 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:11.956321 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:11.956607 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:11.956610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:11.956612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:11.956703 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:11.957105 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:11.957109 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:11.957197 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 06:45:11.957309 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 06:45:11.957441 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for kudu/127.7.48.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
I20260928 06:45:11.980751 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:36049
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:38581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:12.092312 8828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:12.092554 8828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:12.092597 8828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:12.096477 8828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:12.096522 8828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:12.096558 8828 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:45:12.096589 8828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:45:12.101513 8828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:38581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.48.62:36049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:12.102415 8828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:12.103394 8828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:12.112829 8833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:12.112859 8834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:12.113003 8828 server_base.cc:1034] running on GCE node
W20260928 06:45:12.113150 8836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:12.113547 8828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:12.114257 8828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:12.115409 8828 hybrid_clock.cc:648] HybridClock initialized: now 1790577912115379 us; error 44 us; skew 500 ppm
Sep 28 06:45:12 dist-test-slave-twl6 krb5kdc[8431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577912, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:12.118451 8828 init.cc:377] Logged in from keytab as kudu/127.7.48.3@KRBTEST.COM (short username kudu)
I20260928 06:45:12.119269 8828 webserver.cc:460] Webserver started at http://127.7.48.3:37015/ using document root <none> and password file <none>
I20260928 06:45:12.119835 8828 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:12.119946 8828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:12.120180 8828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:12.122159 8828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "2be9c44931f94499952cd211071042ae"
format_stamp: "Formatted at 2026-09-28 06:45:12 on dist-test-slave-twl6"
server_key: "9217b9463610efd13fda68773970527b"
server_key_iv: "aaa1d9a9c1695f5ce10ba9c1eb15b0af"
server_key_version: "encryptionkey@0"
I20260928 06:45:12.122699 8828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "2be9c44931f94499952cd211071042ae"
format_stamp: "Formatted at 2026-09-28 06:45:12 on dist-test-slave-twl6"
server_key: "9217b9463610efd13fda68773970527b"
server_key_iv: "aaa1d9a9c1695f5ce10ba9c1eb15b0af"
server_key_version: "encryptionkey@0"
I20260928 06:45:12.126407 8828 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260928 06:45:12.128684 8843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:12.129793 8828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:12.129927 8828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "2be9c44931f94499952cd211071042ae"
format_stamp: "Formatted at 2026-09-28 06:45:12 on dist-test-slave-twl6"
server_key: "9217b9463610efd13fda68773970527b"
server_key_iv: "aaa1d9a9c1695f5ce10ba9c1eb15b0af"
server_key_version: "encryptionkey@0"
I20260928 06:45:12.130031 8828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:12.144814 8828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:12.148408 8828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:12.148615 8828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:12.150170 8828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:12.150241 8828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:12.150314 8828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:12.150362 8828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:45:12 dist-test-slave-twl6 krb5kdc[8431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577912, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:12.161314 8828 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:43289
I20260928 06:45:12.162411 8828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:45:12.163069 8958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:43289 every 8 connection(s)
I20260928 06:45:12.166064 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.151053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:47993 (local address 127.7.48.62:36049)
0928 06:45:12.151190 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:12.151194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:12.152279 (+ 1085us) server_negotiation.cc:407] Connection header received
0928 06:45:12.153242 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:12.153246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:12.153305 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:12.153370 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:12.155205 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.156106 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.158077 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.158396 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.158480 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.161595 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:12.161598 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:12.161601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:12.161633 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:12.163910 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:12.164549 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:12.164552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:12.164554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:12.164666 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:12.165000 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:12.165003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:12.165005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:12.165200 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:12.165316 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:12.165773 (+ 457us) server_negotiation.cc:299] Negotiation successful
0928 06:45:12.165893 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 06:45:12.166704 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 8828
I20260928 06:45:12.166834 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:45:12.166834 8852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.151611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:36049 (local address 127.7.48.3:47993)
0928 06:45:12.152116 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:12.152161 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:12.152973 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:12.153869 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:12.153881 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:12.154359 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:12.155024 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.155045 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.156266 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.156274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.157907 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.157921 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.158574 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.158580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.158823 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.159470 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:12.159495 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:12.161391 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:12.164055 (+ 2664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:12.164061 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:12.164075 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:12.164450 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:12.164765 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:12.164774 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:12.164776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:12.164897 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:12.165312 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:12.165318 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:12.165547 (+ 229us) client_negotiation.cc:769] Sending connection context
0928 06:45:12.165728 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 06:45:12.166019 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":126,"threads_started":1}
I20260928 06:45:12.167182 7360 external_mini_cluster.cc:1428] Setting key b83d936c1c3ac5fb15f0425d135a7851
I20260928 06:45:12.168143 8959 heartbeater.cc:347] Connected to a master server at 127.7.48.62:36049
I20260928 06:45:12.168277 8959 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:12.168843 8959 heartbeater.cc:511] Master 127.7.48.62:36049 requested a full tablet report, sending...
I20260928 06:45:12.169834 8475 ts_manager.cc:194] Registered new tserver with Master: 2be9c44931f94499952cd211071042ae (127.7.48.3:43289)
I20260928 06:45:12.170471 8475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.3@KRBTEST.COM'} at 127.7.48.3:47993
I20260928 06:45:12.177428 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.170302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:36049 (local address 127.7.48.3:44367)
0928 06:45:12.170705 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:12.170718 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:12.170804 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:12.171985 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:12.171989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:12.172195 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:12.172309 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.172316 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.173021 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.173023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.173599 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.173606 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.173993 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.173996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.174061 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.174477 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:12.174488 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:12.174812 (+ 324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:12.176276 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:12.176279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:12.176280 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:12.176468 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:12.176700 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:12.176702 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:12.176703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:12.176772 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:12.177062 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:12.177064 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:12.177129 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 06:45:12.177201 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 06:45:12.177296 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":117,"threads_started":1}
I20260928 06:45:12.177522 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.170425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:44367 (local address 127.7.48.62:36049)
0928 06:45:12.170544 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:12.170548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:12.170766 (+ 218us) server_negotiation.cc:407] Connection header received
0928 06:45:12.170881 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:12.170884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:12.170923 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:12.171015 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:12.172427 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.172910 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.173700 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.173901 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.173975 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.174926 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:12.174928 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:12.174930 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:12.174950 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:12.176185 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:12.176547 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:12.176549 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:12.176551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:12.176626 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:12.176843 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:12.176845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:12.176846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:12.176988 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:12.177059 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:12.177267 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 06:45:12.177376 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:45:12.180809 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:45:12 dist-test-slave-twl6 krb5kdc[8431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577912, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:45:12 dist-test-slave-twl6 krb5kdc[8431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577912, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:12.205461 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.197959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35848 (local address 127.7.48.62:36049)
0928 06:45:12.198086 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:12.198090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:12.198103 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:45:12.198143 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:12.198145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:12.198181 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:12.198253 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:12.198965 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.199457 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.200260 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.200461 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.200536 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.202783 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:12.202786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:12.202788 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:12.202808 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:12.204102 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:12.204455 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:12.204457 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:12.204458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:12.204529 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:12.204781 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:12.204783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:12.204784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:12.204938 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:12.205035 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:12.205255 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 06:45:12.205340 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 06:45:12.208225 8475 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:45:12.211246 8475 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:45:12.225528 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.220603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45674 (local address 127.7.48.3:43289)
0928 06:45:12.221299 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:12.221308 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:12.221330 (+ 22us) server_negotiation.cc:407] Connection header received
0928 06:45:12.221389 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:12.221393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:12.221542 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:12.221700 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:12.221991 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.222829 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.224260 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.224803 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.224926 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.225104 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:12.225283 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 06:45:12.225376 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":113,"threads_started":1}
I20260928 06:45:12.225518 8976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.220440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:43289 (local address 127.0.0.1:45674)
0928 06:45:12.220816 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:12.220864 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:12.220998 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:12.221694 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:12.221699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:12.221721 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:12.221859 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.221866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.222952 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.222955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.224123 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.224135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.224926 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.224929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.225059 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.225103 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 06:45:12.225236 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 06:45:12.225370 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":5,"spinlock_wait_cycles":6784,"thread_start_us":110,"threads_started":1}
I20260928 06:45:12.227002 8979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.219673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:42719 (local address 127.0.0.1:57366)
0928 06:45:12.220815 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:12.220866 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:12.220999 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:12.221931 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:12.221934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:12.221954 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:12.222105 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.222112 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.223469 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.223472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.225007 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.225019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.226388 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.226391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.226628 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.226677 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 06:45:12.226753 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:45:12.226834 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"mutex_wait_us":252,"spinlock_wait_cycles":7680,"thread_start_us":76,"threads_started":1}
I20260928 06:45:12.227041 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.219816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57366 (local address 127.7.48.1:42719)
0928 06:45:12.221429 (+ 1613us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:12.221435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:12.221454 (+ 19us) server_negotiation.cc:407] Connection header received
0928 06:45:12.221517 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:12.221521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:12.221677 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:12.221793 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:12.222213 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.223329 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.225158 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.226108 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.226212 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.226338 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:12.226807 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 06:45:12.226922 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1518,"thread_start_us":86,"threads_started":1}
I20260928 06:45:12.227317 8978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.220172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52736 (local address 127.7.48.2:45041)
0928 06:45:12.220471 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:12.220483 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:12.221284 (+ 801us) server_negotiation.cc:407] Connection header received
0928 06:45:12.221377 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:12.221382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:12.221578 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:12.221744 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:12.222154 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.223329 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.225155 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.226011 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.226095 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.226259 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:12.227075 (+ 816us) server_negotiation.cc:299] Negotiation successful
0928 06:45:12.227191 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":118,"threads_started":1}
I20260928 06:45:12.227577 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.220468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:45041 (local address 127.0.0.1:52736)
0928 06:45:12.221110 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:12.221124 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:12.221212 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:12.221860 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:12.221862 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:12.221877 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:12.222044 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.222050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.223469 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.223472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.225027 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.225033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.226358 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.226363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.226637 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.226818 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 06:45:12.226898 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 06:45:12.226955 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":303,"thread_start_us":126,"threads_started":1}
I20260928 06:45:12.228616 8893 tablet_service.cc:1467] Processing CreateTablet for tablet bec164ef833e4a4eb39918512f727049 (DEFAULT_TABLE table=test-table [id=e81bd1325ec64289ba1213a779757dbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:12.229667 8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec164ef833e4a4eb39918512f727049. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:12.229880 8753 tablet_service.cc:1467] Processing CreateTablet for tablet bec164ef833e4a4eb39918512f727049 (DEFAULT_TABLE table=test-table [id=e81bd1325ec64289ba1213a779757dbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:12.229868 8613 tablet_service.cc:1467] Processing CreateTablet for tablet bec164ef833e4a4eb39918512f727049 (DEFAULT_TABLE table=test-table [id=e81bd1325ec64289ba1213a779757dbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:12.230813 8613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec164ef833e4a4eb39918512f727049. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:12.230813 8753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec164ef833e4a4eb39918512f727049. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:12.237743 8983 tablet_bootstrap.cc:492] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba: Bootstrap starting.
I20260928 06:45:12.238339 8982 tablet_bootstrap.cc:492] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae: Bootstrap starting.
I20260928 06:45:12.239756 8983 tablet_bootstrap.cc:654] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:12.240082 8982 tablet_bootstrap.cc:654] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:12.240193 8984 tablet_bootstrap.cc:492] T bec164ef833e4a4eb39918512f727049 P 22879aa1a4e3406e87590c3bf4bc96d8: Bootstrap starting.
I20260928 06:45:12.240674 8983 log.cc:824] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:12.240934 8982 log.cc:824] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:12.242123 8984 tablet_bootstrap.cc:654] T bec164ef833e4a4eb39918512f727049 P 22879aa1a4e3406e87590c3bf4bc96d8: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:12.242573 8983 tablet_bootstrap.cc:492] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba: No bootstrap required, opened a new log
I20260928 06:45:12.242775 8983 ts_tablet_manager.cc:1397] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260928 06:45:12.242785 8984 log.cc:824] T bec164ef833e4a4eb39918512f727049 P 22879aa1a4e3406e87590c3bf4bc96d8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:12.244432 8984 tablet_bootstrap.cc:492] T bec164ef833e4a4eb39918512f727049 P 22879aa1a4e3406e87590c3bf4bc96d8: No bootstrap required, opened a new log
I20260928 06:45:12.244634 8984 ts_tablet_manager.cc:1397] T bec164ef833e4a4eb39918512f727049 P 22879aa1a4e3406e87590c3bf4bc96d8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260928 06:45:12.245159 8982 tablet_bootstrap.cc:492] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae: No bootstrap required, opened a new log
I20260928 06:45:12.245340 8982 ts_tablet_manager.cc:1397] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260928 06:45:12.246757 8983 raft_consensus.cc:348] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.247031 8983 raft_consensus.cc:374] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:12.247093 8983 raft_consensus.cc:729] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4526e1b23db04332929ca7d58baac1ba, State: Initialized, Role: FOLLOWER
I20260928 06:45:12.247650 8983 consensus_queue.cc:260] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.248414 8679 heartbeater.cc:503] Master 127.7.48.62:36049 was elected leader, sending a full tablet report...
I20260928 06:45:12.248698 8983 ts_tablet_manager.cc:1428] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260928 06:45:12.249079 8982 raft_consensus.cc:348] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.249265 8982 raft_consensus.cc:374] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:12.249343 8982 raft_consensus.cc:729] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2be9c44931f94499952cd211071042ae, State: Initialized, Role: FOLLOWER
I20260928 06:45:12.248059 8984 raft_consensus.cc:348] T bec164ef833e4a4eb39918512f727049 P 22879aa1a4e3406e87590c3bf4bc96d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.249439 8984 raft_consensus.cc:374] T bec164ef833e4a4eb39918512f727049 P 22879aa1a4e3406e87590c3bf4bc96d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:12.249506 8984 raft_consensus.cc:729] T bec164ef833e4a4eb39918512f727049 P 22879aa1a4e3406e87590c3bf4bc96d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22879aa1a4e3406e87590c3bf4bc96d8, State: Initialized, Role: FOLLOWER
I20260928 06:45:12.249782 8982 consensus_queue.cc:260] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.249964 8984 consensus_queue.cc:260] T bec164ef833e4a4eb39918512f727049 P 22879aa1a4e3406e87590c3bf4bc96d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.250379 8959 heartbeater.cc:503] Master 127.7.48.62:36049 was elected leader, sending a full tablet report...
I20260928 06:45:12.250713 8982 ts_tablet_manager.cc:1428] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 06:45:12.250881 8984 ts_tablet_manager.cc:1428] T bec164ef833e4a4eb39918512f727049 P 22879aa1a4e3406e87590c3bf4bc96d8: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260928 06:45:12.251220 8819 heartbeater.cc:503] Master 127.7.48.62:36049 was elected leader, sending a full tablet report...
I20260928 06:45:12.402163 8988 raft_consensus.cc:482] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:45:12.402406 8988 raft_consensus.cc:504] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.403661 8988 leader_election.cc:288] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22879aa1a4e3406e87590c3bf4bc96d8 (127.7.48.2:45041), 2be9c44931f94499952cd211071042ae (127.7.48.3:43289)
I20260928 06:45:12.404420 8989 raft_consensus.cc:482] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:45:12.404630 8989 raft_consensus.cc:504] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.406797 8989 leader_election.cc:288] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4526e1b23db04332929ca7d58baac1ba (127.7.48.1:42719), 22879aa1a4e3406e87590c3bf4bc96d8 (127.7.48.2:45041)
I20260928 06:45:12.408582 8978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.403997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:59297 (local address 127.7.48.2:45041)
0928 06:45:12.404135 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:12.404140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:12.404357 (+ 217us) server_negotiation.cc:407] Connection header received
0928 06:45:12.404517 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:12.404523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:12.404587 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:12.404663 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:12.405101 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.405632 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.407151 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.407763 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.407877 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.407961 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:12.408316 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 06:45:12.408413 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:45:12.409019 8991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.403914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:45041 (local address 127.7.48.1:59297)
0928 06:45:12.404243 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:12.404257 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:12.404386 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:12.404798 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:12.404802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:12.404822 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:12.404958 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.404965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.405755 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.405762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.407046 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.407058 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.408020 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.408025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.408142 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.408173 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 06:45:12.408707 (+ 534us) client_negotiation.cc:240] Negotiation successful
0928 06:45:12.408784 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":146,"threads_started":1}
I20260928 06:45:12.409797 8773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec164ef833e4a4eb39918512f727049" candidate_uuid: "4526e1b23db04332929ca7d58baac1ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" is_pre_election: true
I20260928 06:45:12.409981 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.404049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:36659 (local address 127.7.48.3:43289)
0928 06:45:12.404266 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:12.404269 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:12.404557 (+ 288us) server_negotiation.cc:407] Connection header received
0928 06:45:12.404707 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:12.404710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:12.404763 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:12.404815 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:12.405169 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.405877 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.407919 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.409319 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.409393 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.409451 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:12.409766 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 06:45:12.409848 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":131}
I20260928 06:45:12.410089 8773 raft_consensus.cc:2454] T bec164ef833e4a4eb39918512f727049 P 22879aa1a4e3406e87590c3bf4bc96d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4526e1b23db04332929ca7d58baac1ba in term 0.
I20260928 06:45:12.410320 8992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.403934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:43289 (local address 127.7.48.1:36659)
0928 06:45:12.404451 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:12.404467 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:12.404566 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:12.404919 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:12.404921 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:12.404935 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:12.405048 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.405055 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.406019 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.406022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.407756 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.407765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.409525 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.409531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.409626 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.409647 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 06:45:12.410097 (+ 450us) client_negotiation.cc:240] Negotiation successful
0928 06:45:12.410175 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":5,"thread_start_us":136,"threads_started":1}
I20260928 06:45:12.410601 8567 leader_election.cc:302] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22879aa1a4e3406e87590c3bf4bc96d8, 4526e1b23db04332929ca7d58baac1ba; no voters:
I20260928 06:45:12.410756 8852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.405916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:42719 (local address 127.7.48.3:59359)
0928 06:45:12.406116 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:12.406129 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:12.406206 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:12.406570 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:12.406573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:12.406589 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:12.406682 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.406688 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.407641 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.407647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.408859 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.408867 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.410361 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.410364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.410467 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.410488 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 06:45:12.410558 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 06:45:12.410649 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":3}
I20260928 06:45:12.410881 8988 raft_consensus.cc:2781] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:45:12.410959 8988 raft_consensus.cc:482] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:45:12.411002 8988 raft_consensus.cc:3037] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:12.411412 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.405990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:59359 (local address 127.7.48.1:42719)
0928 06:45:12.406347 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:12.406351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:12.406362 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:45:12.406396 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:12.406399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:12.406434 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:12.406493 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:12.406796 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.407522 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.409297 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.409897 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.409975 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.410027 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:12.411203 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0928 06:45:12.411271 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":293}
I20260928 06:45:12.411424 8993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.406312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:45041 (local address 127.7.48.3:51495)
0928 06:45:12.407013 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:12.407026 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:12.407115 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:12.407363 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:12.407365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:12.407380 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:12.407474 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.407478 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.408540 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.408543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.409747 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.409757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.410954 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.410959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.411065 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.411087 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 06:45:12.411162 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 06:45:12.411244 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":2,"thread_start_us":124,"threads_started":1}
I20260928 06:45:12.411769 8994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.406657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:51495 (local address 127.7.48.2:45041)
0928 06:45:12.407023 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:12.407035 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:12.407171 (+ 136us) server_negotiation.cc:407] Connection header received
0928 06:45:12.407206 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:12.407208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:12.407258 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:12.407333 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:12.407602 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.408414 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.409885 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.410569 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.410910 (+ 341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.410965 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:12.411562 (+ 597us) server_negotiation.cc:299] Negotiation successful
0928 06:45:12.411646 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":63,"threads_started":1}
I20260928 06:45:12.412030 8633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec164ef833e4a4eb39918512f727049" candidate_uuid: "2be9c44931f94499952cd211071042ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4526e1b23db04332929ca7d58baac1ba" is_pre_election: true
I20260928 06:45:12.412141 8773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec164ef833e4a4eb39918512f727049" candidate_uuid: "2be9c44931f94499952cd211071042ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" is_pre_election: true
I20260928 06:45:12.412124 8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec164ef833e4a4eb39918512f727049" candidate_uuid: "4526e1b23db04332929ca7d58baac1ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2be9c44931f94499952cd211071042ae" is_pre_election: true
I20260928 06:45:12.412253 8773 raft_consensus.cc:2454] T bec164ef833e4a4eb39918512f727049 P 22879aa1a4e3406e87590c3bf4bc96d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2be9c44931f94499952cd211071042ae in term 0.
I20260928 06:45:12.412289 8913 raft_consensus.cc:2454] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4526e1b23db04332929ca7d58baac1ba in term 0.
I20260928 06:45:12.412501 8988 raft_consensus.cc:504] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.412655 8847 leader_election.cc:302] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22879aa1a4e3406e87590c3bf4bc96d8, 2be9c44931f94499952cd211071042ae; no voters:
I20260928 06:45:12.412837 8988 leader_election.cc:288] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [CANDIDATE]: Term 1 election: Requested vote from peers 22879aa1a4e3406e87590c3bf4bc96d8 (127.7.48.2:45041), 2be9c44931f94499952cd211071042ae (127.7.48.3:43289)
I20260928 06:45:12.412941 8633 raft_consensus.cc:2379] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2be9c44931f94499952cd211071042ae in current term 1: Already voted for candidate 4526e1b23db04332929ca7d58baac1ba in this term.
I20260928 06:45:12.413205 8773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec164ef833e4a4eb39918512f727049" candidate_uuid: "4526e1b23db04332929ca7d58baac1ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22879aa1a4e3406e87590c3bf4bc96d8"
I20260928 06:45:12.413233 8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec164ef833e4a4eb39918512f727049" candidate_uuid: "4526e1b23db04332929ca7d58baac1ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2be9c44931f94499952cd211071042ae"
I20260928 06:45:12.413262 8989 raft_consensus.cc:2781] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:45:12.413312 8773 raft_consensus.cc:3037] T bec164ef833e4a4eb39918512f727049 P 22879aa1a4e3406e87590c3bf4bc96d8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:12.413365 8913 raft_consensus.cc:3037] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:12.414748 8913 raft_consensus.cc:2454] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4526e1b23db04332929ca7d58baac1ba in term 1.
I20260928 06:45:12.414860 8989 raft_consensus.cc:482] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:45:12.414959 8989 raft_consensus.cc:3037] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:45:12.415160 8566 leader_election.cc:302] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2be9c44931f94499952cd211071042ae, 4526e1b23db04332929ca7d58baac1ba; no voters:
I20260928 06:45:12.415259 8773 raft_consensus.cc:2454] T bec164ef833e4a4eb39918512f727049 P 22879aa1a4e3406e87590c3bf4bc96d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4526e1b23db04332929ca7d58baac1ba in term 1.
I20260928 06:45:12.415328 8988 raft_consensus.cc:2781] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:12.415549 8988 raft_consensus.cc:686] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [term 1 LEADER]: Becoming Leader. State: Replica: 4526e1b23db04332929ca7d58baac1ba, State: Running, Role: LEADER
W20260928 06:45:12.415684 8960 tablet.cc:2369] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:45:12.415907 8988 consensus_queue.cc:237] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.416116 8989 raft_consensus.cc:504] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.416846 8989 leader_election.cc:288] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae [CANDIDATE]: Term 2 election: Requested vote from peers 4526e1b23db04332929ca7d58baac1ba (127.7.48.1:42719), 22879aa1a4e3406e87590c3bf4bc96d8 (127.7.48.2:45041)
I20260928 06:45:12.417179 8633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec164ef833e4a4eb39918512f727049" candidate_uuid: "2be9c44931f94499952cd211071042ae" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4526e1b23db04332929ca7d58baac1ba"
I20260928 06:45:12.417184 8773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec164ef833e4a4eb39918512f727049" candidate_uuid: "2be9c44931f94499952cd211071042ae" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22879aa1a4e3406e87590c3bf4bc96d8"
I20260928 06:45:12.417296 8773 raft_consensus.cc:3037] T bec164ef833e4a4eb39918512f727049 P 22879aa1a4e3406e87590c3bf4bc96d8 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:45:12.418300 8773 raft_consensus.cc:2454] T bec164ef833e4a4eb39918512f727049 P 22879aa1a4e3406e87590c3bf4bc96d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be9c44931f94499952cd211071042ae in term 2.
I20260928 06:45:12.418576 8847 leader_election.cc:302] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22879aa1a4e3406e87590c3bf4bc96d8, 2be9c44931f94499952cd211071042ae; no voters: 4526e1b23db04332929ca7d58baac1ba
I20260928 06:45:12.418792 8989 raft_consensus.cc:2781] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:45:12.418996 8989 raft_consensus.cc:686] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae [term 2 LEADER]: Becoming Leader. State: Replica: 2be9c44931f94499952cd211071042ae, State: Running, Role: LEADER
I20260928 06:45:12.419250 8989 consensus_queue.cc:237] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.419360 8476 catalog_manager.cc:5495] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 4526e1b23db04332929ca7d58baac1ba (127.7.48.1). New cstate: current_term: 1 leader_uuid: "4526e1b23db04332929ca7d58baac1ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } health_report { overall_health: UNKNOWN } } }
I20260928 06:45:12.421855 8475 catalog_manager.cc:5495] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae reported cstate change: term changed from 1 to 2, leader changed from 4526e1b23db04332929ca7d58baac1ba (127.7.48.1) to 2be9c44931f94499952cd211071042ae (127.7.48.3). New cstate: current_term: 2 leader_uuid: "2be9c44931f94499952cd211071042ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } health_report { overall_health: HEALTHY } } }
W20260928 06:45:12.445497 8820 tablet.cc:2369] T bec164ef833e4a4eb39918512f727049 P 22879aa1a4e3406e87590c3bf4bc96d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:45:12.464354 8680 tablet.cc:2369] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:45:12.471199 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.467328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45676 (local address 127.7.48.3:43289)
0928 06:45:12.467466 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:12.467470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:12.467485 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:45:12.467676 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:12.467680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:12.467743 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:12.467851 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:45:12.468150 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.468729 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.469702 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.469925 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.470027 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.470391 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:45:12.470772 (+ 381us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:45:12.470908 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:12.470983 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 06:45:12.471036 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 06:45:12.476778 8633 raft_consensus.cc:3032] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [term 1 LEADER]: Stepping down as leader of term 1
I20260928 06:45:12.476877 8633 raft_consensus.cc:729] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4526e1b23db04332929ca7d58baac1ba, State: Running, Role: LEADER
I20260928 06:45:12.476984 8773 raft_consensus.cc:1261] T bec164ef833e4a4eb39918512f727049 P 22879aa1a4e3406e87590c3bf4bc96d8 [term 2 FOLLOWER]: Refusing update from remote peer 2be9c44931f94499952cd211071042ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260928 06:45:12.477065 8633 consensus_queue.cc:260] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.477329 8633 raft_consensus.cc:3037] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:45:12.477574 8997 consensus_queue.cc:1035] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:12.478731 8633 raft_consensus.cc:1261] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [term 2 FOLLOWER]: Refusing update from remote peer 2be9c44931f94499952cd211071042ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260928 06:45:12.479185 8997 consensus_queue.cc:1035] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:12.479300 8997 consensus_queue.cc:1225] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae [LEADER]: Peer 4526e1b23db04332929ca7d58baac1ba log is divergent from this leader: its last log entry 1.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260928 06:45:12.480209 8633 pending_rounds.cc:76] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba: Aborting all ops after (but not including) 0
I20260928 06:45:12.480286 8633 pending_rounds.cc:97] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260928 06:45:12.480374 8633 raft_consensus.cc:2866] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260928 06:45:12.483508 9000 mvcc.cc:205] Tried to move back new op lower bound from 7334207129497907200 to 7334207129271496704. Current Snapshot: MvccSnapshot[applied={T|T < 7334207129497907200}]
I20260928 06:45:12.497762 8475 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:12.497937 8475 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:12.500303 8475 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a52b5269fe354aef9616d50180bd5505: Sending DeleteTablet for 3 replicas of tablet bec164ef833e4a4eb39918512f727049
I20260928 06:45:12.501114 8893 tablet_service.cc:1514] Processing DeleteTablet for tablet bec164ef833e4a4eb39918512f727049 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:12 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:45674
I20260928 06:45:12.501116 8753 tablet_service.cc:1514] Processing DeleteTablet for tablet bec164ef833e4a4eb39918512f727049 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:12 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:52736
I20260928 06:45:12.501261 8613 tablet_service.cc:1514] Processing DeleteTablet for tablet bec164ef833e4a4eb39918512f727049 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:12 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:57366
I20260928 06:45:12.501654 9009 tablet_replica.cc:330] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae: stopping tablet replica
I20260928 06:45:12.501876 9011 tablet_replica.cc:330] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba: stopping tablet replica
I20260928 06:45:12.501961 9009 raft_consensus.cc:2229] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae [term 2 LEADER]: Raft consensus shutting down.
I20260928 06:45:12.502275 9011 raft_consensus.cc:2229] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [term 2 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:12.502336 9009 raft_consensus.cc:2258] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae [term 2 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:12.502307 8475 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 06:45:12.502709 9011 raft_consensus.cc:2258] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba [term 2 FOLLOWER]: Raft consensus is shut down!
W20260928 06:45:12.502712 8475 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:45:12.503139 9010 tablet_replica.cc:330] T bec164ef833e4a4eb39918512f727049 P 22879aa1a4e3406e87590c3bf4bc96d8: stopping tablet replica
I20260928 06:45:12.503465 9010 raft_consensus.cc:2229] T bec164ef833e4a4eb39918512f727049 P 22879aa1a4e3406e87590c3bf4bc96d8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:12.503803 9010 raft_consensus.cc:2258] T bec164ef833e4a4eb39918512f727049 P 22879aa1a4e3406e87590c3bf4bc96d8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:12.503820 9009 ts_tablet_manager.cc:1905] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:12.504189 9011 ts_tablet_manager.cc:1905] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:12.505038 9010 ts_tablet_manager.cc:1905] T bec164ef833e4a4eb39918512f727049 P 22879aa1a4e3406e87590c3bf4bc96d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:12.506860 9009 ts_tablet_manager.cc:1918] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20260928 06:45:12.506894 9011 ts_tablet_manager.cc:1918] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20260928 06:45:12.506956 9009 log.cc:1196] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/bec164ef833e4a4eb39918512f727049
I20260928 06:45:12.506984 9011 log.cc:1196] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-0/wal/wals/bec164ef833e4a4eb39918512f727049
I20260928 06:45:12.507294 9009 ts_tablet_manager.cc:1939] T bec164ef833e4a4eb39918512f727049 P 2be9c44931f94499952cd211071042ae: Deleting consensus metadata
I20260928 06:45:12.507313 9011 ts_tablet_manager.cc:1939] T bec164ef833e4a4eb39918512f727049 P 4526e1b23db04332929ca7d58baac1ba: Deleting consensus metadata
I20260928 06:45:12.508174 9010 ts_tablet_manager.cc:1918] T bec164ef833e4a4eb39918512f727049 P 22879aa1a4e3406e87590c3bf4bc96d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20260928 06:45:12.508490 8462 catalog_manager.cc:4841] TS 4526e1b23db04332929ca7d58baac1ba (127.7.48.1:42719): tablet bec164ef833e4a4eb39918512f727049 (table test-table [id=e81bd1325ec64289ba1213a779757dbd]) successfully deleted
I20260928 06:45:12.508682 8462 catalog_manager.cc:4841] TS 2be9c44931f94499952cd211071042ae (127.7.48.3:43289): tablet bec164ef833e4a4eb39918512f727049 (table test-table [id=e81bd1325ec64289ba1213a779757dbd]) successfully deleted
I20260928 06:45:12.508832 9010 log.cc:1196] T bec164ef833e4a4eb39918512f727049 P 22879aa1a4e3406e87590c3bf4bc96d8: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/bec164ef833e4a4eb39918512f727049
I20260928 06:45:12.509191 9010 ts_tablet_manager.cc:1939] T bec164ef833e4a4eb39918512f727049 P 22879aa1a4e3406e87590c3bf4bc96d8: Deleting consensus metadata
I20260928 06:45:12.510192 8463 catalog_manager.cc:4841] TS 22879aa1a4e3406e87590c3bf4bc96d8 (127.7.48.2:45041): tablet bec164ef833e4a4eb39918512f727049 (table test-table [id=e81bd1325ec64289ba1213a779757dbd]) successfully deleted
I20260928 06:45:12.512836 8613 tablet_service.cc:1467] Processing CreateTablet for tablet 3d76bba40eff45ad8dd4e228e2250cf2 (DEFAULT_TABLE table=test-table [id=c6f82848b75941f1ba317340a8e37fcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:12.512991 8753 tablet_service.cc:1467] Processing CreateTablet for tablet 3d76bba40eff45ad8dd4e228e2250cf2 (DEFAULT_TABLE table=test-table [id=c6f82848b75941f1ba317340a8e37fcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:12.512992 8893 tablet_service.cc:1467] Processing CreateTablet for tablet 3d76bba40eff45ad8dd4e228e2250cf2 (DEFAULT_TABLE table=test-table [id=c6f82848b75941f1ba317340a8e37fcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:12.513104 8613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d76bba40eff45ad8dd4e228e2250cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:12.513211 8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d76bba40eff45ad8dd4e228e2250cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:12.513228 8753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d76bba40eff45ad8dd4e228e2250cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:12.515383 8982 tablet_bootstrap.cc:492] T 3d76bba40eff45ad8dd4e228e2250cf2 P 2be9c44931f94499952cd211071042ae: Bootstrap starting.
I20260928 06:45:12.515409 8983 tablet_bootstrap.cc:492] T 3d76bba40eff45ad8dd4e228e2250cf2 P 4526e1b23db04332929ca7d58baac1ba: Bootstrap starting.
I20260928 06:45:12.516259 8983 tablet_bootstrap.cc:654] T 3d76bba40eff45ad8dd4e228e2250cf2 P 4526e1b23db04332929ca7d58baac1ba: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:12.516294 8982 tablet_bootstrap.cc:654] T 3d76bba40eff45ad8dd4e228e2250cf2 P 2be9c44931f94499952cd211071042ae: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:12.516842 8984 tablet_bootstrap.cc:492] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8: Bootstrap starting.
I20260928 06:45:12.517732 8984 tablet_bootstrap.cc:654] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:12.517833 8982 tablet_bootstrap.cc:492] T 3d76bba40eff45ad8dd4e228e2250cf2 P 2be9c44931f94499952cd211071042ae: No bootstrap required, opened a new log
I20260928 06:45:12.517925 8982 ts_tablet_manager.cc:1397] T 3d76bba40eff45ad8dd4e228e2250cf2 P 2be9c44931f94499952cd211071042ae: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 06:45:12.518477 8982 raft_consensus.cc:348] T 3d76bba40eff45ad8dd4e228e2250cf2 P 2be9c44931f94499952cd211071042ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.518571 8982 raft_consensus.cc:374] T 3d76bba40eff45ad8dd4e228e2250cf2 P 2be9c44931f94499952cd211071042ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:12.518599 8982 raft_consensus.cc:729] T 3d76bba40eff45ad8dd4e228e2250cf2 P 2be9c44931f94499952cd211071042ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2be9c44931f94499952cd211071042ae, State: Initialized, Role: FOLLOWER
I20260928 06:45:12.518707 8982 consensus_queue.cc:260] T 3d76bba40eff45ad8dd4e228e2250cf2 P 2be9c44931f94499952cd211071042ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.518944 8982 ts_tablet_manager.cc:1428] T 3d76bba40eff45ad8dd4e228e2250cf2 P 2be9c44931f94499952cd211071042ae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:45:12.519239 8983 tablet_bootstrap.cc:492] T 3d76bba40eff45ad8dd4e228e2250cf2 P 4526e1b23db04332929ca7d58baac1ba: No bootstrap required, opened a new log
I20260928 06:45:12.519310 8983 ts_tablet_manager.cc:1397] T 3d76bba40eff45ad8dd4e228e2250cf2 P 4526e1b23db04332929ca7d58baac1ba: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 06:45:12.519701 8983 raft_consensus.cc:348] T 3d76bba40eff45ad8dd4e228e2250cf2 P 4526e1b23db04332929ca7d58baac1ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.519793 8983 raft_consensus.cc:374] T 3d76bba40eff45ad8dd4e228e2250cf2 P 4526e1b23db04332929ca7d58baac1ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:12.519822 8983 raft_consensus.cc:729] T 3d76bba40eff45ad8dd4e228e2250cf2 P 4526e1b23db04332929ca7d58baac1ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4526e1b23db04332929ca7d58baac1ba, State: Initialized, Role: FOLLOWER
I20260928 06:45:12.519948 8983 consensus_queue.cc:260] T 3d76bba40eff45ad8dd4e228e2250cf2 P 4526e1b23db04332929ca7d58baac1ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.520200 8983 ts_tablet_manager.cc:1428] T 3d76bba40eff45ad8dd4e228e2250cf2 P 4526e1b23db04332929ca7d58baac1ba: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:45:12.526417 8984 tablet_bootstrap.cc:492] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8: No bootstrap required, opened a new log
I20260928 06:45:12.526497 8984 ts_tablet_manager.cc:1397] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20260928 06:45:12.526916 8984 raft_consensus.cc:348] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.527010 8984 raft_consensus.cc:374] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:12.527040 8984 raft_consensus.cc:729] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22879aa1a4e3406e87590c3bf4bc96d8, State: Initialized, Role: FOLLOWER
I20260928 06:45:12.527164 8984 consensus_queue.cc:260] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.527390 8984 ts_tablet_manager.cc:1428] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:45:12.661352 8990 raft_consensus.cc:482] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:45:12.661506 8990 raft_consensus.cc:504] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.662707 8990 leader_election.cc:288] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4526e1b23db04332929ca7d58baac1ba (127.7.48.1:42719), 2be9c44931f94499952cd211071042ae (127.7.48.3:43289)
I20260928 06:45:12.666837 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.663095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:50335 (local address 127.7.48.1:42719)
0928 06:45:12.663250 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:12.663256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:12.663370 (+ 114us) server_negotiation.cc:407] Connection header received
0928 06:45:12.663554 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:12.663557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:12.663609 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:12.663682 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:12.664178 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.664664 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.665773 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.666176 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.666296 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.666359 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:12.666625 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 06:45:12.666707 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:45:12.666926 9016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.663438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:43289 (local address 127.7.48.2:34163)
0928 06:45:12.663747 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:12.663800 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:12.663980 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:12.664253 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:12.664255 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:12.664271 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:12.664379 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.664385 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.665129 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.665132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.665868 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.665876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.666521 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.666527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.666634 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.666654 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 06:45:12.666721 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:45:12.666781 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"spinlock_wait_cycles":11648,"thread_start_us":112,"threads_started":1}
I20260928 06:45:12.666916 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.663726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:34163 (local address 127.7.48.3:43289)
0928 06:45:12.663931 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:12.663935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:12.663950 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:45:12.664103 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:12.664105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:12.664148 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:12.664232 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:12.664524 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.665000 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.665996 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.666410 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.666510 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.666564 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:12.666744 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 06:45:12.666801 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260928 06:45:12.667160 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.662962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:42719 (local address 127.7.48.2:50335)
0928 06:45:12.663250 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:12.663266 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:12.663435 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:12.663831 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:12.663834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:12.663849 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:12.664061 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.664067 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.664775 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.664781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.665651 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.665660 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.666310 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.666314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.666439 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.666493 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 06:45:12.666977 (+ 484us) client_negotiation.cc:240] Negotiation successful
0928 06:45:12.667052 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260928 06:45:12.667510 8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d76bba40eff45ad8dd4e228e2250cf2" candidate_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2be9c44931f94499952cd211071042ae" is_pre_election: true
I20260928 06:45:12.667513 8633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d76bba40eff45ad8dd4e228e2250cf2" candidate_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4526e1b23db04332929ca7d58baac1ba" is_pre_election: true
I20260928 06:45:12.667658 8913 raft_consensus.cc:2454] T 3d76bba40eff45ad8dd4e228e2250cf2 P 2be9c44931f94499952cd211071042ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22879aa1a4e3406e87590c3bf4bc96d8 in term 0.
I20260928 06:45:12.667716 8633 raft_consensus.cc:2454] T 3d76bba40eff45ad8dd4e228e2250cf2 P 4526e1b23db04332929ca7d58baac1ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22879aa1a4e3406e87590c3bf4bc96d8 in term 0.
I20260928 06:45:12.668035 8706 leader_election.cc:302] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22879aa1a4e3406e87590c3bf4bc96d8, 2be9c44931f94499952cd211071042ae; no voters:
I20260928 06:45:12.668318 8990 raft_consensus.cc:2781] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:45:12.668390 8990 raft_consensus.cc:482] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:45:12.668465 8990 raft_consensus.cc:3037] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:12.669499 8990 raft_consensus.cc:504] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.669811 8990 leader_election.cc:288] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8 [CANDIDATE]: Term 1 election: Requested vote from peers 4526e1b23db04332929ca7d58baac1ba (127.7.48.1:42719), 2be9c44931f94499952cd211071042ae (127.7.48.3:43289)
I20260928 06:45:12.670145 8633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d76bba40eff45ad8dd4e228e2250cf2" candidate_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4526e1b23db04332929ca7d58baac1ba"
I20260928 06:45:12.670279 8633 raft_consensus.cc:3037] T 3d76bba40eff45ad8dd4e228e2250cf2 P 4526e1b23db04332929ca7d58baac1ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:12.670248 8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d76bba40eff45ad8dd4e228e2250cf2" candidate_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2be9c44931f94499952cd211071042ae"
I20260928 06:45:12.670361 8913 raft_consensus.cc:3037] T 3d76bba40eff45ad8dd4e228e2250cf2 P 2be9c44931f94499952cd211071042ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:12.671159 8633 raft_consensus.cc:2454] T 3d76bba40eff45ad8dd4e228e2250cf2 P 4526e1b23db04332929ca7d58baac1ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22879aa1a4e3406e87590c3bf4bc96d8 in term 1.
I20260928 06:45:12.671221 8913 raft_consensus.cc:2454] T 3d76bba40eff45ad8dd4e228e2250cf2 P 2be9c44931f94499952cd211071042ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22879aa1a4e3406e87590c3bf4bc96d8 in term 1.
I20260928 06:45:12.671463 8706 leader_election.cc:302] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22879aa1a4e3406e87590c3bf4bc96d8, 4526e1b23db04332929ca7d58baac1ba; no voters:
I20260928 06:45:12.671622 8990 raft_consensus.cc:2781] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:12.671849 8990 raft_consensus.cc:686] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8 [term 1 LEADER]: Becoming Leader. State: Replica: 22879aa1a4e3406e87590c3bf4bc96d8, State: Running, Role: LEADER
I20260928 06:45:12.672133 8990 consensus_queue.cc:237] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.674456 8476 catalog_manager.cc:5495] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22879aa1a4e3406e87590c3bf4bc96d8 (127.7.48.2). New cstate: current_term: 1 leader_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } health_report { overall_health: UNKNOWN } } }
Sep 28 06:45:12 dist-test-slave-twl6 krb5kdc[8431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577911, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:12.749419 9025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.741450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:36049 (local address 127.0.0.1:35862)
0928 06:45:12.741752 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:12.741767 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:12.741879 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:12.742122 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:12.742123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:12.742494 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:12.742603 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.742609 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.743302 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.743305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.743844 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.743850 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.744323 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.744325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.744382 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.744891 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:12.744902 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:12.746226 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:12.747671 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:12.747681 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:12.747687 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:12.748028 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:12.748319 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:12.748323 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:12.748325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:12.748413 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:12.748779 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:12.748787 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:12.748911 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 06:45:12.749027 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 06:45:12.749174 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":103,"threads_started":1}
I20260928 06:45:12.749418 9024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.741494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35862 (local address 127.7.48.62:36049)
0928 06:45:12.741707 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:12.741710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:12.741847 (+ 137us) server_negotiation.cc:407] Connection header received
0928 06:45:12.741956 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:12.741959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:12.741997 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:12.742085 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:12.742714 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.743216 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.743993 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.744228 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.744303 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.746348 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:12.746351 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:12.746353 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:12.746372 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:12.747561 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:12.748142 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:12.748146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:12.748148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:12.748229 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:12.748500 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:12.748505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:12.748507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:12.748674 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:12.748779 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:12.749081 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 06:45:12.749230 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260928 06:45:12.753189 8476 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:35862:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 06:45:12.754011 8476 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:45:12.758788 8613 tablet_service.cc:1467] Processing CreateTablet for tablet dcf81dd965d14bfdb7418b187a7c904d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=14bbaef3f4b44e0a988f20aba9056dd4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:45:12.758914 8893 tablet_service.cc:1467] Processing CreateTablet for tablet dcf81dd965d14bfdb7418b187a7c904d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=14bbaef3f4b44e0a988f20aba9056dd4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:45:12.758878 8753 tablet_service.cc:1467] Processing CreateTablet for tablet dcf81dd965d14bfdb7418b187a7c904d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=14bbaef3f4b44e0a988f20aba9056dd4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:45:12.759232 8613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcf81dd965d14bfdb7418b187a7c904d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:12.759250 8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcf81dd965d14bfdb7418b187a7c904d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:12.759271 8753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcf81dd965d14bfdb7418b187a7c904d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:12.762005 8982 tablet_bootstrap.cc:492] T dcf81dd965d14bfdb7418b187a7c904d P 2be9c44931f94499952cd211071042ae: Bootstrap starting.
I20260928 06:45:12.762068 8983 tablet_bootstrap.cc:492] T dcf81dd965d14bfdb7418b187a7c904d P 4526e1b23db04332929ca7d58baac1ba: Bootstrap starting.
I20260928 06:45:12.762913 8983 tablet_bootstrap.cc:654] T dcf81dd965d14bfdb7418b187a7c904d P 4526e1b23db04332929ca7d58baac1ba: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:12.762913 8982 tablet_bootstrap.cc:654] T dcf81dd965d14bfdb7418b187a7c904d P 2be9c44931f94499952cd211071042ae: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:12.763490 8984 tablet_bootstrap.cc:492] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8: Bootstrap starting.
I20260928 06:45:12.764250 8982 tablet_bootstrap.cc:492] T dcf81dd965d14bfdb7418b187a7c904d P 2be9c44931f94499952cd211071042ae: No bootstrap required, opened a new log
I20260928 06:45:12.764338 8982 ts_tablet_manager.cc:1397] T dcf81dd965d14bfdb7418b187a7c904d P 2be9c44931f94499952cd211071042ae: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:12.764405 8983 tablet_bootstrap.cc:492] T dcf81dd965d14bfdb7418b187a7c904d P 4526e1b23db04332929ca7d58baac1ba: No bootstrap required, opened a new log
I20260928 06:45:12.764475 8983 ts_tablet_manager.cc:1397] T dcf81dd965d14bfdb7418b187a7c904d P 4526e1b23db04332929ca7d58baac1ba: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:12.764700 8984 tablet_bootstrap.cc:654] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:12.764756 8982 raft_consensus.cc:348] T dcf81dd965d14bfdb7418b187a7c904d P 2be9c44931f94499952cd211071042ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.764864 8982 raft_consensus.cc:374] T dcf81dd965d14bfdb7418b187a7c904d P 2be9c44931f94499952cd211071042ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:12.764901 8982 raft_consensus.cc:729] T dcf81dd965d14bfdb7418b187a7c904d P 2be9c44931f94499952cd211071042ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2be9c44931f94499952cd211071042ae, State: Initialized, Role: FOLLOWER
I20260928 06:45:12.764869 8983 raft_consensus.cc:348] T dcf81dd965d14bfdb7418b187a7c904d P 4526e1b23db04332929ca7d58baac1ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.764961 8983 raft_consensus.cc:374] T dcf81dd965d14bfdb7418b187a7c904d P 4526e1b23db04332929ca7d58baac1ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:12.764992 8983 raft_consensus.cc:729] T dcf81dd965d14bfdb7418b187a7c904d P 4526e1b23db04332929ca7d58baac1ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4526e1b23db04332929ca7d58baac1ba, State: Initialized, Role: FOLLOWER
I20260928 06:45:12.765023 8982 consensus_queue.cc:260] T dcf81dd965d14bfdb7418b187a7c904d P 2be9c44931f94499952cd211071042ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.765107 8983 consensus_queue.cc:260] T dcf81dd965d14bfdb7418b187a7c904d P 4526e1b23db04332929ca7d58baac1ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.765285 8982 ts_tablet_manager.cc:1428] T dcf81dd965d14bfdb7418b187a7c904d P 2be9c44931f94499952cd211071042ae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:45:12.765372 8983 ts_tablet_manager.cc:1428] T dcf81dd965d14bfdb7418b187a7c904d P 4526e1b23db04332929ca7d58baac1ba: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:45:12.766074 8984 tablet_bootstrap.cc:492] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8: No bootstrap required, opened a new log
I20260928 06:45:12.766163 8984 ts_tablet_manager.cc:1397] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:45:12.766149 9005 tablet_replica.cc:439] T dcf81dd965d14bfdb7418b187a7c904d P 2be9c44931f94499952cd211071042ae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } } }
I20260928 06:45:12.766525 9005 tablet_replica.cc:442] T dcf81dd965d14bfdb7418b187a7c904d P 2be9c44931f94499952cd211071042ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:45:12.766572 8984 raft_consensus.cc:348] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.766678 8984 raft_consensus.cc:374] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:12.766710 8984 raft_consensus.cc:729] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22879aa1a4e3406e87590c3bf4bc96d8, State: Initialized, Role: FOLLOWER
I20260928 06:45:12.766825 8984 consensus_queue.cc:260] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.767088 8984 ts_tablet_manager.cc:1428] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:45:12.767350 8990 tablet_replica.cc:439] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } } }
I20260928 06:45:12.767464 8990 tablet_replica.cc:442] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:45:12.768507 8988 tablet_replica.cc:439] T dcf81dd965d14bfdb7418b187a7c904d P 4526e1b23db04332929ca7d58baac1ba: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } } }
I20260928 06:45:12.768646 8988 tablet_replica.cc:442] T dcf81dd965d14bfdb7418b187a7c904d P 4526e1b23db04332929ca7d58baac1ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:45:12.799607 8990 raft_consensus.cc:482] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:45:12.799750 8990 raft_consensus.cc:504] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.800239 8990 leader_election.cc:288] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4526e1b23db04332929ca7d58baac1ba (127.7.48.1:42719), 2be9c44931f94499952cd211071042ae (127.7.48.3:43289)
I20260928 06:45:12.800697 8633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcf81dd965d14bfdb7418b187a7c904d" candidate_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4526e1b23db04332929ca7d58baac1ba" is_pre_election: true
I20260928 06:45:12.800740 8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcf81dd965d14bfdb7418b187a7c904d" candidate_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2be9c44931f94499952cd211071042ae" is_pre_election: true
I20260928 06:45:12.800825 8633 raft_consensus.cc:2454] T dcf81dd965d14bfdb7418b187a7c904d P 4526e1b23db04332929ca7d58baac1ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22879aa1a4e3406e87590c3bf4bc96d8 in term 0.
I20260928 06:45:12.800931 8913 raft_consensus.cc:2454] T dcf81dd965d14bfdb7418b187a7c904d P 2be9c44931f94499952cd211071042ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22879aa1a4e3406e87590c3bf4bc96d8 in term 0.
I20260928 06:45:12.801175 8706 leader_election.cc:302] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22879aa1a4e3406e87590c3bf4bc96d8, 4526e1b23db04332929ca7d58baac1ba; no voters:
I20260928 06:45:12.801373 8990 raft_consensus.cc:2781] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:45:12.801445 8990 raft_consensus.cc:482] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:45:12.801497 8990 raft_consensus.cc:3037] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:12.802448 8990 raft_consensus.cc:504] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.802760 8990 leader_election.cc:288] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8 [CANDIDATE]: Term 1 election: Requested vote from peers 4526e1b23db04332929ca7d58baac1ba (127.7.48.1:42719), 2be9c44931f94499952cd211071042ae (127.7.48.3:43289)
I20260928 06:45:12.803117 8633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcf81dd965d14bfdb7418b187a7c904d" candidate_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4526e1b23db04332929ca7d58baac1ba"
I20260928 06:45:12.803224 8633 raft_consensus.cc:3037] T dcf81dd965d14bfdb7418b187a7c904d P 4526e1b23db04332929ca7d58baac1ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:12.803233 8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcf81dd965d14bfdb7418b187a7c904d" candidate_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2be9c44931f94499952cd211071042ae"
I20260928 06:45:12.803354 8913 raft_consensus.cc:3037] T dcf81dd965d14bfdb7418b187a7c904d P 2be9c44931f94499952cd211071042ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:12.804306 8913 raft_consensus.cc:2454] T dcf81dd965d14bfdb7418b187a7c904d P 2be9c44931f94499952cd211071042ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22879aa1a4e3406e87590c3bf4bc96d8 in term 1.
I20260928 06:45:12.804584 8633 raft_consensus.cc:2454] T dcf81dd965d14bfdb7418b187a7c904d P 4526e1b23db04332929ca7d58baac1ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22879aa1a4e3406e87590c3bf4bc96d8 in term 1.
I20260928 06:45:12.804628 8706 leader_election.cc:302] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22879aa1a4e3406e87590c3bf4bc96d8, 2be9c44931f94499952cd211071042ae; no voters:
I20260928 06:45:12.804800 8990 raft_consensus.cc:2781] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:12.804891 8990 raft_consensus.cc:686] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8 [term 1 LEADER]: Becoming Leader. State: Replica: 22879aa1a4e3406e87590c3bf4bc96d8, State: Running, Role: LEADER
I20260928 06:45:12.805024 8990 consensus_queue.cc:237] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } }
I20260928 06:45:12.805707 9017 tablet_replica.cc:439] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8: TxnStatusTablet state changed. Reason: New leader 22879aa1a4e3406e87590c3bf4bc96d8. Latest consensus state: current_term: 1 leader_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } } }
I20260928 06:45:12.805794 9017 tablet_replica.cc:442] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:45:12.806126 9030 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 06:45:12.806299 9030 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 06:45:12.806668 8475 catalog_manager.cc:5495] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22879aa1a4e3406e87590c3bf4bc96d8 (127.7.48.2). New cstate: current_term: 1 leader_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } health_report { overall_health: UNKNOWN } } }
I20260928 06:45:12.819701 9025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.815614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:45041 (local address 127.0.0.1:52744)
0928 06:45:12.815742 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:12.815765 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:12.815847 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:12.816229 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:12.816231 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:12.816239 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:45:12.816375 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.816380 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.817385 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.817390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.818065 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.818074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.818684 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.818686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.818747 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.818788 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:45:12.819471 (+ 683us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:45:12.819481 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 06:45:12.819530 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:45:12.819598 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 06:45:12.819717 8994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.815676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52744 (local address 127.7.48.2:45041)
0928 06:45:12.815793 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:12.815797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:12.815813 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:45:12.816033 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:12.816039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:12.816100 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:12.816211 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:45:12.816458 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.817265 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.818188 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.818575 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.818683 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.818920 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:45:12.819381 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:45:12.819490 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:12.819543 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 06:45:12.819598 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:45:12.822242 8633 raft_consensus.cc:1261] T dcf81dd965d14bfdb7418b187a7c904d P 4526e1b23db04332929ca7d58baac1ba [term 1 FOLLOWER]: Refusing update from remote peer 22879aa1a4e3406e87590c3bf4bc96d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:12.822484 8913 raft_consensus.cc:1261] T dcf81dd965d14bfdb7418b187a7c904d P 2be9c44931f94499952cd211071042ae [term 1 FOLLOWER]: Refusing update from remote peer 22879aa1a4e3406e87590c3bf4bc96d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:12.822700 8990 consensus_queue.cc:1035] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:12.822983 9017 consensus_queue.cc:1035] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:12.824299 8988 tablet_replica.cc:439] T dcf81dd965d14bfdb7418b187a7c904d P 4526e1b23db04332929ca7d58baac1ba: TxnStatusTablet state changed. Reason: New leader 22879aa1a4e3406e87590c3bf4bc96d8. Latest consensus state: current_term: 1 leader_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } } }
I20260928 06:45:12.824419 8988 tablet_replica.cc:442] T dcf81dd965d14bfdb7418b187a7c904d P 4526e1b23db04332929ca7d58baac1ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:45:12.824637 9005 tablet_replica.cc:439] T dcf81dd965d14bfdb7418b187a7c904d P 2be9c44931f94499952cd211071042ae: TxnStatusTablet state changed. Reason: New leader 22879aa1a4e3406e87590c3bf4bc96d8. Latest consensus state: current_term: 1 leader_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } } }
I20260928 06:45:12.824745 9005 tablet_replica.cc:442] T dcf81dd965d14bfdb7418b187a7c904d P 2be9c44931f94499952cd211071042ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:45:12.825977 9017 tablet_replica.cc:439] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } } }
I20260928 06:45:12.826097 9017 tablet_replica.cc:442] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:45:12.826817 9001 mvcc.cc:205] Tried to move back new op lower bound from 7334207130915168256 to 7334207130850664448. Current Snapshot: MvccSnapshot[applied={T|T < 7334207130915168256 or (T in {7334207130915168256})}]
I20260928 06:45:12.827384 8988 tablet_replica.cc:439] T dcf81dd965d14bfdb7418b187a7c904d P 4526e1b23db04332929ca7d58baac1ba: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } } }
I20260928 06:45:12.827466 9003 mvcc.cc:205] Tried to move back new op lower bound from 7334207130915168256 to 7334207130850664448. Current Snapshot: MvccSnapshot[applied={T|T < 7334207130915168256}]
I20260928 06:45:12.828492 8990 tablet_replica.cc:439] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } } }
I20260928 06:45:12.828598 8990 tablet_replica.cc:442] T dcf81dd965d14bfdb7418b187a7c904d P 22879aa1a4e3406e87590c3bf4bc96d8: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:45:12.827476 8988 tablet_replica.cc:442] T dcf81dd965d14bfdb7418b187a7c904d P 4526e1b23db04332929ca7d58baac1ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:45:12.829191 9005 tablet_replica.cc:439] T dcf81dd965d14bfdb7418b187a7c904d P 2be9c44931f94499952cd211071042ae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 } } peers { permanent_uuid: "22879aa1a4e3406e87590c3bf4bc96d8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45041 } } peers { permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 } } }
I20260928 06:45:12.829298 9005 tablet_replica.cc:442] T dcf81dd965d14bfdb7418b187a7c904d P 2be9c44931f94499952cd211071042ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:45:12.836663 8994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.833301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52746 (local address 127.7.48.2:45041)
0928 06:45:12.833699 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:12.833703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:12.833715 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:45:12.833753 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:12.833756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:12.833798 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:12.833867 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:45:12.834106 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.834872 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.835610 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.835967 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.836060 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.836280 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:45:12.836355 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:45:12.836422 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:12.836496 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 06:45:12.836542 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325}
I20260928 06:45:12.846796 8994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.843282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:41859 (local address 127.7.48.2:45041)
0928 06:45:12.843398 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:12.843402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:12.843628 (+ 226us) server_negotiation.cc:407] Connection header received
0928 06:45:12.843736 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:12.843738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:12.843774 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:12.843831 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:45:12.844108 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.844539 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.845809 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.846001 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.846073 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.846399 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:45:12.846455 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:45:12.846507 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:12.846628 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 06:45:12.846680 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":46}
I20260928 06:45:12.846910 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.843171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:45041 (local address 127.7.48.2:41859)
0928 06:45:12.843538 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:12.843551 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:12.843647 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:12.843860 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:12.843863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:12.843870 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:45:12.844010 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.844016 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.844659 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.844663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.845678 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:12.845688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.846105 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.846109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:12.846227 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.846277 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:45:12.846537 (+ 260us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:45:12.846543 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:45:12.846607 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 06:45:12.846712 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":19,"thread_start_us":136,"threads_started":1}
I20260928 06:45:12.853484 8633 raft_consensus.cc:1261] T 3d76bba40eff45ad8dd4e228e2250cf2 P 4526e1b23db04332929ca7d58baac1ba [term 1 FOLLOWER]: Refusing update from remote peer 22879aa1a4e3406e87590c3bf4bc96d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:12.853539 8913 raft_consensus.cc:1261] T 3d76bba40eff45ad8dd4e228e2250cf2 P 2be9c44931f94499952cd211071042ae [term 1 FOLLOWER]: Refusing update from remote peer 22879aa1a4e3406e87590c3bf4bc96d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:12.853897 9017 consensus_queue.cc:1035] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be9c44931f94499952cd211071042ae" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:12.854027 8990 consensus_queue.cc:1035] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4526e1b23db04332929ca7d58baac1ba" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:12.889317 8475 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:12.889452 8475 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:12.891353 8475 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a52b5269fe354aef9616d50180bd5505: Sending DeleteTablet for 3 replicas of tablet 3d76bba40eff45ad8dd4e228e2250cf2
I20260928 06:45:12.891932 8753 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d76bba40eff45ad8dd4e228e2250cf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:12 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:52736
I20260928 06:45:12.891947 8613 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d76bba40eff45ad8dd4e228e2250cf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:12 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:57366
I20260928 06:45:12.892107 9010 tablet_replica.cc:330] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8: stopping tablet replica
I20260928 06:45:12.892112 9011 tablet_replica.cc:330] T 3d76bba40eff45ad8dd4e228e2250cf2 P 4526e1b23db04332929ca7d58baac1ba: stopping tablet replica
I20260928 06:45:12.892210 9010 raft_consensus.cc:2229] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:45:12.892215 9011 raft_consensus.cc:2229] T 3d76bba40eff45ad8dd4e228e2250cf2 P 4526e1b23db04332929ca7d58baac1ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:12.892311 9011 raft_consensus.cc:2258] T 3d76bba40eff45ad8dd4e228e2250cf2 P 4526e1b23db04332929ca7d58baac1ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:12.892468 9010 raft_consensus.cc:2258] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:12.892864 8893 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d76bba40eff45ad8dd4e228e2250cf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:12 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:45674
I20260928 06:45:12.893270 9010 ts_tablet_manager.cc:1905] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:12.893728 9009 tablet_replica.cc:330] T 3d76bba40eff45ad8dd4e228e2250cf2 P 2be9c44931f94499952cd211071042ae: stopping tablet replica
I20260928 06:45:12.893858 9009 raft_consensus.cc:2229] T 3d76bba40eff45ad8dd4e228e2250cf2 P 2be9c44931f94499952cd211071042ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:12.893992 9009 raft_consensus.cc:2258] T 3d76bba40eff45ad8dd4e228e2250cf2 P 2be9c44931f94499952cd211071042ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:12.895063 9009 ts_tablet_manager.cc:1905] T 3d76bba40eff45ad8dd4e228e2250cf2 P 2be9c44931f94499952cd211071042ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:12.895063 9011 ts_tablet_manager.cc:1905] T 3d76bba40eff45ad8dd4e228e2250cf2 P 4526e1b23db04332929ca7d58baac1ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:12.896056 9010 ts_tablet_manager.cc:1918] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:45:12.896121 9010 log.cc:1196] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/3d76bba40eff45ad8dd4e228e2250cf2
I20260928 06:45:12.896374 9010 ts_tablet_manager.cc:1939] T 3d76bba40eff45ad8dd4e228e2250cf2 P 22879aa1a4e3406e87590c3bf4bc96d8: Deleting consensus metadata
I20260928 06:45:12.896965 8463 catalog_manager.cc:4841] TS 22879aa1a4e3406e87590c3bf4bc96d8 (127.7.48.2:45041): tablet 3d76bba40eff45ad8dd4e228e2250cf2 (table test-table [id=c6f82848b75941f1ba317340a8e37fcb]) successfully deleted
I20260928 06:45:12.897632 9009 ts_tablet_manager.cc:1918] T 3d76bba40eff45ad8dd4e228e2250cf2 P 2be9c44931f94499952cd211071042ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:45:12.897632 9011 ts_tablet_manager.cc:1918] T 3d76bba40eff45ad8dd4e228e2250cf2 P 4526e1b23db04332929ca7d58baac1ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:45:12.897699 9009 log.cc:1196] T 3d76bba40eff45ad8dd4e228e2250cf2 P 2be9c44931f94499952cd211071042ae: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/3d76bba40eff45ad8dd4e228e2250cf2
I20260928 06:45:12.897699 9011 log.cc:1196] T 3d76bba40eff45ad8dd4e228e2250cf2 P 4526e1b23db04332929ca7d58baac1ba: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577905526996-7360-0/minicluster-data/ts-0/wal/wals/3d76bba40eff45ad8dd4e228e2250cf2
I20260928 06:45:12.897974 9011 ts_tablet_manager.cc:1939] T 3d76bba40eff45ad8dd4e228e2250cf2 P 4526e1b23db04332929ca7d58baac1ba: Deleting consensus metadata
I20260928 06:45:12.898518 9009 ts_tablet_manager.cc:1939] T 3d76bba40eff45ad8dd4e228e2250cf2 P 2be9c44931f94499952cd211071042ae: Deleting consensus metadata
I20260928 06:45:12.898645 8462 catalog_manager.cc:4841] TS 4526e1b23db04332929ca7d58baac1ba (127.7.48.1:42719): tablet 3d76bba40eff45ad8dd4e228e2250cf2 (table test-table [id=c6f82848b75941f1ba317340a8e37fcb]) successfully deleted
I20260928 06:45:12.899205 8462 catalog_manager.cc:4841] TS 2be9c44931f94499952cd211071042ae (127.7.48.3:43289): tablet 3d76bba40eff45ad8dd4e228e2250cf2 (table test-table [id=c6f82848b75941f1ba317340a8e37fcb]) successfully deleted
Sep 28 06:45:12 dist-test-slave-twl6 krb5kdc[8431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577912, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.48.1@KRBTEST.COM
I20260928 06:45:12.907718 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.893615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57382 (local address 127.7.48.1:42719)
0928 06:45:12.894198 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:12.894202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:12.894217 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:45:12.894252 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:12.894255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:12.894303 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:12.894393 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:12.899213 (+ 4820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.899782 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.900623 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.900960 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.901064 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.904249 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:12.904259 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:12.904274 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:12.904298 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:12.906009 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:12.906450 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:12.906460 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:12.906462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:12.906562 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:12.906819 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:12.906823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:12.906824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:12.907102 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:12.907313 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:12.907432 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 06:45:12.907549 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489}
W20260928 06:45:12.908138 8673 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57382
I20260928 06:45:12.917454 9024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:12.909836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35866 (local address 127.7.48.62:36049)
0928 06:45:12.909941 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:12.909945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:12.909995 (+ 50us) server_negotiation.cc:407] Connection header received
0928 06:45:12.910098 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:12.910101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:12.910437 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:12.910520 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:12.911161 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.911794 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.912608 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:12.912851 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:12.912955 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:12.914233 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:12.914237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:12.914240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:12.914272 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:12.915556 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:12.916080 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:12.916083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:12.916084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:12.916153 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:12.916430 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:12.916433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:12.916435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:12.916734 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:12.916902 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:12.917118 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 06:45:12.917239 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 06:45:12.917905 8475 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35866
I20260928 06:45:12.919093 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 8548
I20260928 06:45:12.929119 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 8688
I20260928 06:45:12.935855 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 8828
I20260928 06:45:12.942523 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 8444
2026-09-28T06:45:12Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2823 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:45:12 dist-test-slave-twl6 krb5kdc[9049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:45:12 dist-test-slave-twl6 krb5kdc[9049](info): set up 2 sockets
Sep 28 06:45:12 dist-test-slave-twl6 krb5kdc[9049](info): commencing operation
krb5kdc: starting...
W20260928 06:45:13.960157 7360 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:45:14 dist-test-slave-twl6 krb5kdc[9049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:45:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:45:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:45:14.075853 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:44081
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:44081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:14.184289 9062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:14.184698 9062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:14.184780 9062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:14.188655 9062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:45:14.188730 9062 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 06:45:14.188759 9062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:14.188802 9062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:14.188834 9062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:45:14.188864 9062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:45:14.194053 9062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:44081
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:44081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:14.195097 9062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:14.196157 9062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:14.204370 9070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:14.204351 9068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:14.204414 9062 server_base.cc:1034] running on GCE node
W20260928 06:45:14.204351 9067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:14.205219 9062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:14.206236 9062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:14.207418 9062 hybrid_clock.cc:648] HybridClock initialized: now 1790577914207403 us; error 34 us; skew 500 ppm
Sep 28 06:45:14 dist-test-slave-twl6 krb5kdc[9049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577914, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:14.210152 9062 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:45:14.211088 9062 webserver.cc:460] Webserver started at http://127.7.48.62:38269/ using document root <none> and password file <none>
I20260928 06:45:14.211617 9062 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:14.211675 9062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:14.211833 9062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:14.213918 9062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "1f4ac496c93b4f55b7c160bfa30fcad7"
format_stamp: "Formatted at 2026-09-28 06:45:14 on dist-test-slave-twl6"
server_key: "f37496c1c4ec3deeb8637d73a3afccbe"
server_key_iv: "bcc436f1531c2fe4897af926f5bc4226"
server_key_version: "encryptionkey@0"
I20260928 06:45:14.214416 9062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "1f4ac496c93b4f55b7c160bfa30fcad7"
format_stamp: "Formatted at 2026-09-28 06:45:14 on dist-test-slave-twl6"
server_key: "f37496c1c4ec3deeb8637d73a3afccbe"
server_key_iv: "bcc436f1531c2fe4897af926f5bc4226"
server_key_version: "encryptionkey@0"
I20260928 06:45:14.217981 9062 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 06:45:14.220297 9077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:14.221473 9062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:45:14.221611 9062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "1f4ac496c93b4f55b7c160bfa30fcad7"
format_stamp: "Formatted at 2026-09-28 06:45:14 on dist-test-slave-twl6"
server_key: "f37496c1c4ec3deeb8637d73a3afccbe"
server_key_iv: "bcc436f1531c2fe4897af926f5bc4226"
server_key_version: "encryptionkey@0"
I20260928 06:45:14.221709 9062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:14.231127 9062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:14.233855 9062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:14.234045 9062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:14.241524 9062 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:44081
I20260928 06:45:14.241564 9139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:44081 every 8 connection(s)
I20260928 06:45:14.242447 9062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:45:14.245121 9140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:14.250284 9140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7: Bootstrap starting.
I20260928 06:45:14.251202 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 9062
I20260928 06:45:14.251295 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:45:14.251564 7360 external_mini_cluster.cc:1428] Setting key d95ebcebeec617c4924957598985e694
I20260928 06:45:14.252959 9140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:14.253690 9140 log.cc:824] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:14.255483 9140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7: No bootstrap required, opened a new log
I20260928 06:45:14.259413 9140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4ac496c93b4f55b7c160bfa30fcad7" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 44081 } }
I20260928 06:45:14.259661 9140 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:14.259734 9140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f4ac496c93b4f55b7c160bfa30fcad7, State: Initialized, Role: FOLLOWER
Sep 28 06:45:14 dist-test-slave-twl6 krb5kdc[9049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:14.260234 9140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4ac496c93b4f55b7c160bfa30fcad7" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 44081 } }
I20260928 06:45:14.260394 9140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:45:14.260465 9140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:45:14.260571 9140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:14.261512 9140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4ac496c93b4f55b7c160bfa30fcad7" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 44081 } }
I20260928 06:45:14.261899 9140 leader_election.cc:302] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f4ac496c93b4f55b7c160bfa30fcad7; no voters:
I20260928 06:45:14.262253 9140 leader_election.cc:288] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:45:14.262387 9145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:14.262591 9145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7 [term 1 LEADER]: Becoming Leader. State: Replica: 1f4ac496c93b4f55b7c160bfa30fcad7, State: Running, Role: LEADER
I20260928 06:45:14.262924 9145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4ac496c93b4f55b7c160bfa30fcad7" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 44081 } }
I20260928 06:45:14.263624 9140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:45:14.264106 9145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f4ac496c93b4f55b7c160bfa30fcad7. Latest consensus state: current_term: 1 leader_uuid: "1f4ac496c93b4f55b7c160bfa30fcad7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4ac496c93b4f55b7c160bfa30fcad7" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 44081 } } }
I20260928 06:45:14.264228 9145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:14.264528 9147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f4ac496c93b4f55b7c160bfa30fcad7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4ac496c93b4f55b7c160bfa30fcad7" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 44081 } } }
I20260928 06:45:14.264680 9147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:14.264631 9149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:45:14.266981 9143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:14.252655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38146 (local address 127.7.48.62:44081)
0928 06:45:14.253124 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:14.253139 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:14.253172 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:45:14.253829 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:14.253850 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:14.254238 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:14.254624 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:14.255386 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.256414 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.257245 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.257712 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.257886 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:14.260865 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:14.260872 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:14.260887 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:14.260916 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:14.263142 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:14.263631 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:14.263637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:14.263642 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:14.263777 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:14.264146 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:14.264149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:14.264151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:14.264901 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:14.265114 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:14.265475 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 06:45:14.265818 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":130,"threads_started":1}
I20260928 06:45:14.269366 9149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:45:14.274839 9149 catalog_manager.cc:1347] Generated new cluster ID: 0b28e741ffea489f9dbc35022806c63b
I20260928 06:45:14.274888 9149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:45:14.281688 9149 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:45:14.282893 9149 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:45:14.288363 9149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7: Generated new TSK 0
I20260928 06:45:14.289165 9149 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
I20260928 06:45:14.318327 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:44081
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:14.429081 9166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:14.429322 9166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:14.429366 9166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:14.433396 9166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:14.433465 9166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:14.433531 9166 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 06:45:14.433573 9166 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:45:14.433655 9166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:45:14.438657 9166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.48.62:44081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:14.439661 9166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:14.440744 9166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:14.450335 9172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:14.450423 9171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:14.450484 9166 server_base.cc:1034] running on GCE node
W20260928 06:45:14.450402 9174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:14.451026 9166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:14.451609 9166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:14.452768 9166 hybrid_clock.cc:648] HybridClock initialized: now 1790577914452737 us; error 47 us; skew 500 ppm
Sep 28 06:45:14 dist-test-slave-twl6 krb5kdc[9049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577914, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:14.455364 9166 init.cc:377] Logged in from keytab as kudu/127.7.48.1@KRBTEST.COM (short username kudu)
I20260928 06:45:14.456282 9166 webserver.cc:460] Webserver started at http://127.7.48.1:46025/ using document root <none> and password file <none>
I20260928 06:45:14.456817 9166 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:14.456903 9166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:14.457140 9166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:14.459079 9166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "e3f506d848e746fa8c27d0e3c46b8379"
format_stamp: "Formatted at 2026-09-28 06:45:14 on dist-test-slave-twl6"
server_key: "ae330139f03ab047f805e9e56feca72e"
server_key_iv: "2d6a5746bb988df615ee32616ceb8a0d"
server_key_version: "encryptionkey@0"
I20260928 06:45:14.459622 9166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "e3f506d848e746fa8c27d0e3c46b8379"
format_stamp: "Formatted at 2026-09-28 06:45:14 on dist-test-slave-twl6"
server_key: "ae330139f03ab047f805e9e56feca72e"
server_key_iv: "2d6a5746bb988df615ee32616ceb8a0d"
server_key_version: "encryptionkey@0"
I20260928 06:45:14.463231 9166 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:45:14.465945 9181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:14.467053 9166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:14.467201 9166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "e3f506d848e746fa8c27d0e3c46b8379"
format_stamp: "Formatted at 2026-09-28 06:45:14 on dist-test-slave-twl6"
server_key: "ae330139f03ab047f805e9e56feca72e"
server_key_iv: "2d6a5746bb988df615ee32616ceb8a0d"
server_key_version: "encryptionkey@0"
I20260928 06:45:14.467304 9166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:14.476210 9166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:14.478648 9166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:14.478832 9166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:14.480324 9166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:14.480393 9166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:14.480458 9166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:14.480512 9166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:45:14 dist-test-slave-twl6 krb5kdc[9049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577914, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:14.490950 9166 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:46737
I20260928 06:45:14.491029 9296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:46737 every 8 connection(s)
I20260928 06:45:14.491860 9166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:45:14.494217 9143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:14.480975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:45795 (local address 127.7.48.62:44081)
0928 06:45:14.481132 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:14.481137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:14.482382 (+ 1245us) server_negotiation.cc:407] Connection header received
0928 06:45:14.483485 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:14.483490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:14.483559 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:14.483675 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:14.484981 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.485718 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.487450 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.487796 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.487923 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:14.490797 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:14.490800 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:14.490802 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:14.490827 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:14.492507 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:14.493021 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:14.493023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:14.493024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:14.493110 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:14.493338 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:14.493340 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:14.493341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:14.493508 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:14.493579 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:14.494000 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 06:45:14.494103 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 06:45:14.494773 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 9166
I20260928 06:45:14.494899 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
I20260928 06:45:14.494966 9190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:14.481797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:44081 (local address 127.7.48.1:45795)
0928 06:45:14.482226 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:14.482273 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:14.483301 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:14.483844 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:14.483852 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:14.484269 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:14.484834 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:14.484847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.485858 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.485864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:14.487310 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:14.487324 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.487966 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.487973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:14.488163 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:14.488722 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:14.488746 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:14.490602 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:14.492639 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:14.492644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:14.492657 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:14.492926 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:14.493175 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:14.493177 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:14.493179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:14.493283 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:14.493591 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:14.493595 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:14.493794 (+ 199us) client_negotiation.cc:769] Sending connection context
0928 06:45:14.493948 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 06:45:14.494178 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":129,"threads_started":1}
I20260928 06:45:14.495224 7360 external_mini_cluster.cc:1428] Setting key 84192b13da109a6dd22fc3cf45c68d04
I20260928 06:45:14.496455 9297 heartbeater.cc:347] Connected to a master server at 127.7.48.62:44081
I20260928 06:45:14.496616 9297 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:14.497182 9297 heartbeater.cc:511] Master 127.7.48.62:44081 requested a full tablet report, sending...
I20260928 06:45:14.498843 9092 ts_manager.cc:194] Registered new tserver with Master: e3f506d848e746fa8c27d0e3c46b8379 (127.7.48.1:46737)
I20260928 06:45:14.500305 9092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.1@KRBTEST.COM'} at 127.7.48.1:45795
I20260928 06:45:14.505659 9143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:14.498695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:48287 (local address 127.7.48.62:44081)
0928 06:45:14.498807 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:14.498811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:14.498956 (+ 145us) server_negotiation.cc:407] Connection header received
0928 06:45:14.499035 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:14.499038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:14.499081 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:14.499150 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:14.499601 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.500175 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.501009 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.501295 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.501397 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:14.502778 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:14.502780 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:14.502782 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:14.502813 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:14.504155 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:14.504599 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:14.504602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:14.504603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:14.504667 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:14.504967 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:14.504970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:14.504971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:14.505108 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:14.505178 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:14.505468 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 06:45:14.505549 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":46}
I20260928 06:45:14.505779 9304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:14.498568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:44081 (local address 127.7.48.1:48287)
0928 06:45:14.498848 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:14.498863 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:14.498963 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:14.499189 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:14.499192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:14.499383 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:14.499490 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:14.499496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.500295 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.500299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:14.500893 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:14.500901 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.501421 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.501427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:14.501633 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:14.502142 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:14.502162 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:14.502630 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:14.504255 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:14.504261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:14.504265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:14.504506 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:14.504749 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:14.504755 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:14.504759 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:14.504878 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:14.505200 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:14.505204 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:14.505300 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 06:45:14.505414 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 06:45:14.505556 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":1,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.7.48.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
I20260928 06:45:14.524196 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:44081
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:14.632247 9306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:14.632486 9306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:14.632532 9306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:14.636431 9306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:14.636480 9306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:14.636518 9306 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 06:45:14.636531 9306 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:45:14.636562 9306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:45:14.641394 9306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.48.62:44081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:14.642313 9306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:14.643271 9306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:14.652765 9314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:14.652746 9311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:14.652770 9312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:14.653134 9306 server_base.cc:1034] running on GCE node
I20260928 06:45:14.653640 9306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:14.654296 9306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:14.655514 9306 hybrid_clock.cc:648] HybridClock initialized: now 1790577914655508 us; error 42 us; skew 500 ppm
Sep 28 06:45:14 dist-test-slave-twl6 krb5kdc[9049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577914, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:14.658360 9306 init.cc:377] Logged in from keytab as kudu/127.7.48.2@KRBTEST.COM (short username kudu)
I20260928 06:45:14.659277 9306 webserver.cc:460] Webserver started at http://127.7.48.2:42847/ using document root <none> and password file <none>
I20260928 06:45:14.659842 9306 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:14.659972 9306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:14.660211 9306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:14.662238 9306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "830f1c5539634e0b9645855927156696"
format_stamp: "Formatted at 2026-09-28 06:45:14 on dist-test-slave-twl6"
server_key: "d2158849fb4ba7978ddad88da2e262b1"
server_key_iv: "a3c397bd2bcc27c05060657f5af15572"
server_key_version: "encryptionkey@0"
I20260928 06:45:14.662819 9306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "830f1c5539634e0b9645855927156696"
format_stamp: "Formatted at 2026-09-28 06:45:14 on dist-test-slave-twl6"
server_key: "d2158849fb4ba7978ddad88da2e262b1"
server_key_iv: "a3c397bd2bcc27c05060657f5af15572"
server_key_version: "encryptionkey@0"
I20260928 06:45:14.666898 9306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260928 06:45:14.669329 9321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:14.670564 9306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:45:14.670729 9306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "830f1c5539634e0b9645855927156696"
format_stamp: "Formatted at 2026-09-28 06:45:14 on dist-test-slave-twl6"
server_key: "d2158849fb4ba7978ddad88da2e262b1"
server_key_iv: "a3c397bd2bcc27c05060657f5af15572"
server_key_version: "encryptionkey@0"
I20260928 06:45:14.670835 9306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:14.688983 9306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:14.691671 9306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:14.691869 9306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:14.693284 9306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:14.693351 9306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:14.693387 9306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:14.693456 9306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:45:14 dist-test-slave-twl6 krb5kdc[9049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577914, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:14.704703 9306 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:32827
I20260928 06:45:14.704773 9436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:32827 every 8 connection(s)
I20260928 06:45:14.705756 9306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:45:14.707595 9143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:14.694188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:33719 (local address 127.7.48.62:44081)
0928 06:45:14.694316 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:14.694320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:14.695319 (+ 999us) server_negotiation.cc:407] Connection header received
0928 06:45:14.696417 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:14.696423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:14.696492 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:14.696561 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:14.697866 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.698602 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.700011 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.700237 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.700348 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:14.703756 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:14.703760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:14.703762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:14.703796 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:14.705542 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:14.706085 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:14.706089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:14.706091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:14.706199 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:14.706688 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:14.706691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:14.706692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:14.706858 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:14.706935 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:14.707368 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 06:45:14.707463 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 06:45:14.708405 9330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:14.694711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:44081 (local address 127.7.48.2:33719)
0928 06:45:14.695182 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:14.695219 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:14.696167 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:14.696962 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:14.696967 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:14.697292 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:14.697719 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:14.697731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.698719 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.698722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:14.699869 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:14.699919 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.700380 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.700384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:14.700554 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:14.701116 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:14.701145 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:14.703573 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:14.705667 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:14.705672 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:14.705682 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:14.705981 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:14.706315 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:14.706318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:14.706321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:14.706486 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:14.706958 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:14.706963 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:14.707166 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 06:45:14.707316 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 06:45:14.707557 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":131,"threads_started":1}
I20260928 06:45:14.709622 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 9306
I20260928 06:45:14.709769 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:45:14.710052 7360 external_mini_cluster.cc:1428] Setting key f83fa263d1618dbda7f0f2a788c8489b
I20260928 06:45:14.711555 9437 heartbeater.cc:347] Connected to a master server at 127.7.48.62:44081
I20260928 06:45:14.711717 9437 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:14.712204 9437 heartbeater.cc:511] Master 127.7.48.62:44081 requested a full tablet report, sending...
I20260928 06:45:14.713151 9092 ts_manager.cc:194] Registered new tserver with Master: 830f1c5539634e0b9645855927156696 (127.7.48.2:32827)
I20260928 06:45:14.713985 9092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.2@KRBTEST.COM'} at 127.7.48.2:33719
I20260928 06:45:14.719365 9143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:14.711784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:45803 (local address 127.7.48.62:44081)
0928 06:45:14.711910 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:14.711914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:14.712377 (+ 463us) server_negotiation.cc:407] Connection header received
0928 06:45:14.712493 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:14.712495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:14.712535 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:14.712618 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:14.713215 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.713763 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.714861 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.715080 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.715181 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:14.716397 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:14.716400 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:14.716403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:14.716425 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:14.717793 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:14.718202 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:14.718205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:14.718207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:14.718287 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:14.718556 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:14.718561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:14.718563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:14.718756 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:14.718858 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:14.719113 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 06:45:14.719217 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:45:14.719386 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:14.712001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:44081 (local address 127.7.48.2:45803)
0928 06:45:14.712294 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:14.712312 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:14.712415 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:14.712653 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:14.712659 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:14.712901 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:14.713065 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:14.713075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.713894 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.713896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:14.714745 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:14.714754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.715185 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.715188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:14.715346 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:14.715868 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:14.715930 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:14.716258 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:14.717902 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:14.717907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:14.717909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:14.718104 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:14.718381 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:14.718383 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:14.718385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:14.718469 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:14.718859 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:14.718863 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:14.718958 (+ 95us) client_negotiation.cc:769] Sending connection context
0928 06:45:14.719063 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 06:45:14.719211 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":4,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.7.48.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
I20260928 06:45:14.736966 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:44081
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:14.845053 9446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:14.845299 9446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:14.845371 9446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:14.849129 9446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:14.849210 9446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:14.849282 9446 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 06:45:14.849324 9446 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:45:14.849375 9446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:45:14.854471 9446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.48.62:44081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:14.855453 9446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:14.856448 9446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:14.865860 9454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:14.865855 9451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:14.865855 9452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:14.866019 9446 server_base.cc:1034] running on GCE node
I20260928 06:45:14.866583 9446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:14.867116 9446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:14.868269 9446 hybrid_clock.cc:648] HybridClock initialized: now 1790577914868250 us; error 30 us; skew 500 ppm
Sep 28 06:45:14 dist-test-slave-twl6 krb5kdc[9049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577914, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:14.870914 9446 init.cc:377] Logged in from keytab as kudu/127.7.48.3@KRBTEST.COM (short username kudu)
I20260928 06:45:14.871815 9446 webserver.cc:460] Webserver started at http://127.7.48.3:39649/ using document root <none> and password file <none>
I20260928 06:45:14.872376 9446 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:14.872438 9446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:14.872665 9446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:14.874644 9446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "387ce6f9361a439f95ede8adcce0c411"
format_stamp: "Formatted at 2026-09-28 06:45:14 on dist-test-slave-twl6"
server_key: "0fa66c88f1ef30fab8907e4973af1b11"
server_key_iv: "da92ee2818a2bca8789d9307e99efe53"
server_key_version: "encryptionkey@0"
I20260928 06:45:14.875255 9446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "387ce6f9361a439f95ede8adcce0c411"
format_stamp: "Formatted at 2026-09-28 06:45:14 on dist-test-slave-twl6"
server_key: "0fa66c88f1ef30fab8907e4973af1b11"
server_key_iv: "da92ee2818a2bca8789d9307e99efe53"
server_key_version: "encryptionkey@0"
I20260928 06:45:14.878938 9446 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 06:45:14.881300 9461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:14.882468 9446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:45:14.882587 9446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "387ce6f9361a439f95ede8adcce0c411"
format_stamp: "Formatted at 2026-09-28 06:45:14 on dist-test-slave-twl6"
server_key: "0fa66c88f1ef30fab8907e4973af1b11"
server_key_iv: "da92ee2818a2bca8789d9307e99efe53"
server_key_version: "encryptionkey@0"
I20260928 06:45:14.882686 9446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:14.895205 9446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:14.897969 9446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:14.898164 9446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:14.899484 9446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:14.899559 9446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:14.899633 9446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:14.899690 9446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:45:14 dist-test-slave-twl6 krb5kdc[9049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577914, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:14.910408 9446 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:34929
I20260928 06:45:14.910445 9576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:34929 every 8 connection(s)
I20260928 06:45:14.911235 9446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:45:14.912066 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 9446
I20260928 06:45:14.912184 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:45:14.912492 7360 external_mini_cluster.cc:1428] Setting key 258c46a2dbc51ad092ba54635985313b
I20260928 06:45:14.914122 9143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:14.900298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:33689 (local address 127.7.48.62:44081)
0928 06:45:14.900421 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:14.900425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:14.901431 (+ 1006us) server_negotiation.cc:407] Connection header received
0928 06:45:14.902546 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:14.902550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:14.902621 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:14.902684 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:14.904398 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.905271 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.906909 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.907287 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.907402 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:14.910103 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:14.910107 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:14.910110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:14.910152 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:14.911757 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:14.912410 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:14.912413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:14.912414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:14.912499 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:14.912926 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:14.912929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:14.912931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:14.913165 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:14.913266 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:14.913831 (+ 565us) server_negotiation.cc:299] Negotiation successful
0928 06:45:14.913962 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:45:14.914939 9470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:14.900814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:44081 (local address 127.7.48.3:33689)
0928 06:45:14.901282 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:14.901319 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:14.902290 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:14.903122 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:14.903139 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:14.903581 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:14.904222 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:14.904241 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.905427 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.905431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:14.906790 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:14.906799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.907431 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.907434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:14.907558 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:14.908100 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:14.908115 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:14.909925 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:14.911917 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:14.911925 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:14.911938 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:14.912293 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:14.912623 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:14.912626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:14.912629 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:14.912795 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:14.913287 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:14.913293 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:14.913576 (+ 283us) client_negotiation.cc:769] Sending connection context
0928 06:45:14.913754 (+ 178us) client_negotiation.cc:240] Negotiation successful
0928 06:45:14.914029 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":141,"threads_started":1}
I20260928 06:45:14.916080 9577 heartbeater.cc:347] Connected to a master server at 127.7.48.62:44081
I20260928 06:45:14.916236 9577 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:14.916677 9577 heartbeater.cc:511] Master 127.7.48.62:44081 requested a full tablet report, sending...
I20260928 06:45:14.917637 9093 ts_manager.cc:194] Registered new tserver with Master: 387ce6f9361a439f95ede8adcce0c411 (127.7.48.3:34929)
I20260928 06:45:14.918331 9093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.3@KRBTEST.COM'} at 127.7.48.3:33689
I20260928 06:45:14.924274 9143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:14.918071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:47697 (local address 127.7.48.62:44081)
0928 06:45:14.918175 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:14.918178 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:14.918316 (+ 138us) server_negotiation.cc:407] Connection header received
0928 06:45:14.918380 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:14.918383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:14.918426 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:14.918503 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:14.918976 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.919439 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.920383 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.920575 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.920651 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:14.921521 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:14.921524 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:14.921526 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:14.921546 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:14.922922 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:14.923317 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:14.923319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:14.923320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:14.923389 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:14.923625 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:14.923627 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:14.923628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:14.923768 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:14.923837 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:14.924088 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 06:45:14.924179 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:45:14.924319 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:14.917991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:44081 (local address 127.7.48.3:47697)
0928 06:45:14.918230 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:14.918244 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:14.918322 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:14.918615 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:14.918618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:14.918765 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:14.918879 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:14.918887 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.919567 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.919570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:14.920285 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:14.920294 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.920666 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.920668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:14.920730 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:14.921116 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:14.921126 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:14.921392 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:14.923015 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:14.923017 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:14.923019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:14.923221 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:14.923477 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:14.923479 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:14.923480 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:14.923553 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:14.923843 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:14.923845 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:14.923965 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 06:45:14.924051 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:45:14.924143 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":94,"threads_started":1}
I20260928 06:45:14.926373 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:45:14 dist-test-slave-twl6 krb5kdc[9049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:45:14 dist-test-slave-twl6 krb5kdc[9049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:14.950237 9143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:14.942698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59792 (local address 127.7.48.62:44081)
0928 06:45:14.942828 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:14.942831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:14.942841 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:45:14.942871 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:14.942873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:14.942904 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:14.942988 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:14.943637 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.944185 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.944986 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.945196 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.945296 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:14.947393 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:14.947403 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:14.947406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:14.947425 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:14.948670 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:14.949068 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:14.949072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:14.949074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:14.949157 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:14.949424 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:14.949427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:14.949429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:14.949587 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:14.949736 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:14.949965 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 06:45:14.950077 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 06:45:14.952943 9092 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:45:14.956008 9092 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:45:14.969866 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:14.963847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:34929 (local address 127.0.0.1:36558)
0928 06:45:14.964195 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:14.964416 (+ 221us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:14.964579 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:14.965258 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:14.965263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:14.965289 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:14.965579 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:14.965589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.967014 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.967017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:14.968447 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:14.968456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.969308 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.969312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:14.969423 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:14.969478 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 06:45:14.969581 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 06:45:14.969696 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":27,"thread_start_us":101,"threads_started":1}
I20260928 06:45:14.970070 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:14.964845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43950 (local address 127.7.48.2:32827)
0928 06:45:14.965306 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:14.965318 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:14.965338 (+ 20us) server_negotiation.cc:407] Connection header received
0928 06:45:14.965410 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:14.965414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:14.965601 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:14.966026 (+ 425us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:14.966155 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.967184 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.968638 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.969412 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.969580 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:14.969731 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:14.969825 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 06:45:14.969925 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":110,"threads_started":1}
I20260928 06:45:14.970381 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:14.964029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36558 (local address 127.7.48.3:34929)
0928 06:45:14.964391 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:14.964405 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:14.964518 (+ 113us) server_negotiation.cc:407] Connection header received
0928 06:45:14.964726 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:14.964732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:14.964939 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:14.965098 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:14.965720 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.966894 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.968579 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.969190 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.969294 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:14.969442 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:14.970174 (+ 732us) server_negotiation.cc:299] Negotiation successful
0928 06:45:14.970277 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":115,"threads_started":1}
I20260928 06:45:14.970347 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:14.964638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:32827 (local address 127.0.0.1:43950)
0928 06:45:14.964879 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:14.964894 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:14.964976 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:14.965750 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:14.965756 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:14.965776 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:14.965862 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:14.965867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.967289 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.967293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:14.968496 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:14.968508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.969592 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.969596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:14.969689 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:14.969711 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 06:45:14.969799 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:45:14.969873 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":59,"thread_start_us":120,"threads_started":1}
I20260928 06:45:14.970805 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:14.965733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59014 (local address 127.7.48.1:46737)
0928 06:45:14.966064 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:14.966076 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:14.966093 (+ 17us) server_negotiation.cc:407] Connection header received
0928 06:45:14.966157 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:14.966160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:14.966274 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:14.966439 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:14.966639 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.967650 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.969091 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.969825 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.969923 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:14.970480 (+ 557us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:14.970591 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 06:45:14.970677 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":88,"threads_started":1}
I20260928 06:45:14.970521 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:14.964972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:46737 (local address 127.0.0.1:59014)
0928 06:45:14.965168 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:14.965181 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:14.965276 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:14.966418 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:14.966421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:14.966438 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:14.966517 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:14.966523 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.967770 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.967773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:14.968957 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:14.968966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:14.970072 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:14.970075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:14.970177 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:14.970201 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 06:45:14.970260 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:45:14.970318 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":32,"thread_start_us":85,"threads_started":1}
I20260928 06:45:14.973588 9511 tablet_service.cc:1467] Processing CreateTablet for tablet de7a9595979546b8a0cda9be393b69f6 (DEFAULT_TABLE table=test-table [id=dd0aae8553db4f6986a45e235d3a0803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:14.973693 9371 tablet_service.cc:1467] Processing CreateTablet for tablet de7a9595979546b8a0cda9be393b69f6 (DEFAULT_TABLE table=test-table [id=dd0aae8553db4f6986a45e235d3a0803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:14.974016 9231 tablet_service.cc:1467] Processing CreateTablet for tablet de7a9595979546b8a0cda9be393b69f6 (DEFAULT_TABLE table=test-table [id=dd0aae8553db4f6986a45e235d3a0803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:14.974640 9371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de7a9595979546b8a0cda9be393b69f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:14.974695 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de7a9595979546b8a0cda9be393b69f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:14.974876 9231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de7a9595979546b8a0cda9be393b69f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:14.981675 9600 tablet_bootstrap.cc:492] T de7a9595979546b8a0cda9be393b69f6 P 830f1c5539634e0b9645855927156696: Bootstrap starting.
I20260928 06:45:14.983655 9600 tablet_bootstrap.cc:654] T de7a9595979546b8a0cda9be393b69f6 P 830f1c5539634e0b9645855927156696: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:14.984256 9601 tablet_bootstrap.cc:492] T de7a9595979546b8a0cda9be393b69f6 P 387ce6f9361a439f95ede8adcce0c411: Bootstrap starting.
I20260928 06:45:14.984601 9600 log.cc:824] T de7a9595979546b8a0cda9be393b69f6 P 830f1c5539634e0b9645855927156696: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:14.985292 9602 tablet_bootstrap.cc:492] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379: Bootstrap starting.
I20260928 06:45:14.986588 9601 tablet_bootstrap.cc:654] T de7a9595979546b8a0cda9be393b69f6 P 387ce6f9361a439f95ede8adcce0c411: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:14.986721 9600 tablet_bootstrap.cc:492] T de7a9595979546b8a0cda9be393b69f6 P 830f1c5539634e0b9645855927156696: No bootstrap required, opened a new log
I20260928 06:45:14.986912 9600 ts_tablet_manager.cc:1397] T de7a9595979546b8a0cda9be393b69f6 P 830f1c5539634e0b9645855927156696: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 06:45:14.987397 9601 log.cc:824] T de7a9595979546b8a0cda9be393b69f6 P 387ce6f9361a439f95ede8adcce0c411: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:14.987536 9602 tablet_bootstrap.cc:654] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:14.988360 9602 log.cc:824] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:14.989378 9601 tablet_bootstrap.cc:492] T de7a9595979546b8a0cda9be393b69f6 P 387ce6f9361a439f95ede8adcce0c411: No bootstrap required, opened a new log
I20260928 06:45:14.989562 9601 ts_tablet_manager.cc:1397] T de7a9595979546b8a0cda9be393b69f6 P 387ce6f9361a439f95ede8adcce0c411: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 06:45:14.990289 9602 tablet_bootstrap.cc:492] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379: No bootstrap required, opened a new log
I20260928 06:45:14.990468 9602 ts_tablet_manager.cc:1397] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 06:45:14.990594 9600 raft_consensus.cc:348] T de7a9595979546b8a0cda9be393b69f6 P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } }
I20260928 06:45:14.990832 9600 raft_consensus.cc:374] T de7a9595979546b8a0cda9be393b69f6 P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:14.990885 9600 raft_consensus.cc:729] T de7a9595979546b8a0cda9be393b69f6 P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830f1c5539634e0b9645855927156696, State: Initialized, Role: FOLLOWER
I20260928 06:45:14.991317 9600 consensus_queue.cc:260] T de7a9595979546b8a0cda9be393b69f6 P 830f1c5539634e0b9645855927156696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } }
I20260928 06:45:14.992013 9437 heartbeater.cc:503] Master 127.7.48.62:44081 was elected leader, sending a full tablet report...
I20260928 06:45:14.992336 9600 ts_tablet_manager.cc:1428] T de7a9595979546b8a0cda9be393b69f6 P 830f1c5539634e0b9645855927156696: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260928 06:45:14.993368 9601 raft_consensus.cc:348] T de7a9595979546b8a0cda9be393b69f6 P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } }
I20260928 06:45:14.993637 9601 raft_consensus.cc:374] T de7a9595979546b8a0cda9be393b69f6 P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:14.993695 9601 raft_consensus.cc:729] T de7a9595979546b8a0cda9be393b69f6 P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 387ce6f9361a439f95ede8adcce0c411, State: Initialized, Role: FOLLOWER
I20260928 06:45:14.994068 9601 consensus_queue.cc:260] T de7a9595979546b8a0cda9be393b69f6 P 387ce6f9361a439f95ede8adcce0c411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } }
I20260928 06:45:14.994729 9577 heartbeater.cc:503] Master 127.7.48.62:44081 was elected leader, sending a full tablet report...
I20260928 06:45:14.995064 9601 ts_tablet_manager.cc:1428] T de7a9595979546b8a0cda9be393b69f6 P 387ce6f9361a439f95ede8adcce0c411: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 06:45:14.997507 9602 raft_consensus.cc:348] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } }
I20260928 06:45:14.997833 9602 raft_consensus.cc:374] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:14.997901 9602 raft_consensus.cc:729] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f506d848e746fa8c27d0e3c46b8379, State: Initialized, Role: FOLLOWER
I20260928 06:45:14.998278 9602 consensus_queue.cc:260] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } }
I20260928 06:45:14.998854 9297 heartbeater.cc:503] Master 127.7.48.62:44081 was elected leader, sending a full tablet report...
I20260928 06:45:14.999133 9602 ts_tablet_manager.cc:1428] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
W20260928 06:45:15.162421 9578 tablet.cc:2369] T de7a9595979546b8a0cda9be393b69f6 P 387ce6f9361a439f95ede8adcce0c411: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:45:15.174598 9608 raft_consensus.cc:482] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:45:15.174846 9608 raft_consensus.cc:504] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } }
I20260928 06:45:15.176352 9608 leader_election.cc:288] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 387ce6f9361a439f95ede8adcce0c411 (127.7.48.3:34929), 830f1c5539634e0b9645855927156696 (127.7.48.2:32827)
I20260928 06:45:15.180540 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:15.177174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:40291 (local address 127.7.48.2:32827)
0928 06:45:15.177299 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:15.177305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:15.177431 (+ 126us) server_negotiation.cc:407] Connection header received
0928 06:45:15.177515 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:15.177518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:15.177564 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:15.177645 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:15.177813 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.178517 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.179501 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.179982 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.180061 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:15.180142 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:15.180343 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 06:45:15.180414 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:45:15.180572 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:15.176729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:38063 (local address 127.7.48.3:34929)
0928 06:45:15.176869 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:15.176876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:15.176955 (+ 79us) server_negotiation.cc:407] Connection header received
0928 06:45:15.177189 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:15.177193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:15.177248 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:15.177323 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:15.177735 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.178522 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.179561 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.180007 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.180128 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:15.180188 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:15.180398 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 06:45:15.180474 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:45:15.180570 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:15.177018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:32827 (local address 127.7.48.1:40291)
0928 06:45:15.177350 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:15.177361 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:15.177444 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:15.177652 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:15.177653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:15.177662 (+ 9us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:15.177747 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:15.177750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.178616 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.178618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:15.179377 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:15.179384 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.180096 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.180101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:15.180185 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:15.180228 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 06:45:15.180313 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 06:45:15.180396 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":94,"threads_started":1}
I20260928 06:45:15.180563 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:15.176581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:34929 (local address 127.7.48.1:38063)
0928 06:45:15.176885 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:15.176898 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:15.177027 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:15.177475 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:15.177479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:15.177499 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:15.177604 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:15.177612 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.178659 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.178661 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:15.179388 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:15.179396 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.180142 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.180146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:15.180254 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:15.180271 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 06:45:15.180370 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 06:45:15.180432 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":129,"threads_started":1}
I20260928 06:45:15.181140 9391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7a9595979546b8a0cda9be393b69f6" candidate_uuid: "e3f506d848e746fa8c27d0e3c46b8379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830f1c5539634e0b9645855927156696" is_pre_election: true
I20260928 06:45:15.181439 9391 raft_consensus.cc:2454] T de7a9595979546b8a0cda9be393b69f6 P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3f506d848e746fa8c27d0e3c46b8379 in term 0.
I20260928 06:45:15.181478 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7a9595979546b8a0cda9be393b69f6" candidate_uuid: "e3f506d848e746fa8c27d0e3c46b8379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "387ce6f9361a439f95ede8adcce0c411" is_pre_election: true
I20260928 06:45:15.181794 9531 raft_consensus.cc:2454] T de7a9595979546b8a0cda9be393b69f6 P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3f506d848e746fa8c27d0e3c46b8379 in term 0.
I20260928 06:45:15.181924 9184 leader_election.cc:302] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830f1c5539634e0b9645855927156696, e3f506d848e746fa8c27d0e3c46b8379; no voters:
I20260928 06:45:15.182231 9608 raft_consensus.cc:2781] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:45:15.182303 9608 raft_consensus.cc:482] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:45:15.182332 9608 raft_consensus.cc:3037] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:15.183358 9608 raft_consensus.cc:504] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } }
I20260928 06:45:15.183681 9608 leader_election.cc:288] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379 [CANDIDATE]: Term 1 election: Requested vote from peers 387ce6f9361a439f95ede8adcce0c411 (127.7.48.3:34929), 830f1c5539634e0b9645855927156696 (127.7.48.2:32827)
I20260928 06:45:15.184116 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7a9595979546b8a0cda9be393b69f6" candidate_uuid: "e3f506d848e746fa8c27d0e3c46b8379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "387ce6f9361a439f95ede8adcce0c411"
I20260928 06:45:15.184124 9391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7a9595979546b8a0cda9be393b69f6" candidate_uuid: "e3f506d848e746fa8c27d0e3c46b8379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830f1c5539634e0b9645855927156696"
I20260928 06:45:15.184247 9531 raft_consensus.cc:3037] T de7a9595979546b8a0cda9be393b69f6 P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:15.184278 9391 raft_consensus.cc:3037] T de7a9595979546b8a0cda9be393b69f6 P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:15.185722 9391 raft_consensus.cc:2454] T de7a9595979546b8a0cda9be393b69f6 P 830f1c5539634e0b9645855927156696 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3f506d848e746fa8c27d0e3c46b8379 in term 1.
I20260928 06:45:15.186024 9184 leader_election.cc:302] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830f1c5539634e0b9645855927156696, e3f506d848e746fa8c27d0e3c46b8379; no voters:
I20260928 06:45:15.186143 9531 raft_consensus.cc:2454] T de7a9595979546b8a0cda9be393b69f6 P 387ce6f9361a439f95ede8adcce0c411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3f506d848e746fa8c27d0e3c46b8379 in term 1.
I20260928 06:45:15.186213 9608 raft_consensus.cc:2781] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:15.186431 9608 raft_consensus.cc:686] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379 [term 1 LEADER]: Becoming Leader. State: Replica: e3f506d848e746fa8c27d0e3c46b8379, State: Running, Role: LEADER
I20260928 06:45:15.186785 9608 consensus_queue.cc:237] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } }
I20260928 06:45:15.189913 9093 catalog_manager.cc:5495] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3f506d848e746fa8c27d0e3c46b8379 (127.7.48.1). New cstate: current_term: 1 leader_uuid: "e3f506d848e746fa8c27d0e3c46b8379" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } health_report { overall_health: HEALTHY } } }
W20260928 06:45:15.206871 9438 tablet.cc:2369] T de7a9595979546b8a0cda9be393b69f6 P 830f1c5539634e0b9645855927156696: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 06:45:15 dist-test-slave-twl6 krb5kdc[9049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577914, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
W20260928 06:45:15.243332 9298 tablet.cc:2369] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:45:15.246541 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:15.237658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:44081 (local address 127.0.0.1:59800)
0928 06:45:15.237952 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:15.237963 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:15.238100 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:15.238338 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:15.238340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:15.238745 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:15.238836 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:15.238843 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.239576 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.239579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:15.240196 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:15.240207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.240682 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.240685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:15.240757 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:15.241279 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:15.241294 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:15.242962 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:15.244956 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:15.244963 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:15.244969 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:15.245250 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:15.245535 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:15.245538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:15.245540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:15.245627 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:15.245973 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:15.245978 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:15.246096 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 06:45:15.246207 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 06:45:15.246350 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":107,"threads_started":1}
I20260928 06:45:15.246551 9143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:15.237701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59800 (local address 127.7.48.62:44081)
0928 06:45:15.237833 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:15.237840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:15.238038 (+ 198us) server_negotiation.cc:407] Connection header received
0928 06:45:15.238209 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:15.238211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:15.238245 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:15.238322 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:15.238953 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.239460 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.240308 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.240576 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.240680 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:15.243104 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:15.243107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:15.243128 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:15.243159 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:15.244848 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:15.245355 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:15.245358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:15.245359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:15.245434 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:15.245713 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:15.245716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:15.245717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:15.245878 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:15.245958 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:15.246253 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 06:45:15.246420 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":38}
I20260928 06:45:15.250784 9093 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:59800:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 06:45:15.251441 9093 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:45:15.256263 9371 tablet_service.cc:1467] Processing CreateTablet for tablet 4a966fda8cd24551b2c7142fc0471c2a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3bbb5bcbf08544a8af28f07cf5d0710e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:45:15.256335 9511 tablet_service.cc:1467] Processing CreateTablet for tablet 4a966fda8cd24551b2c7142fc0471c2a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3bbb5bcbf08544a8af28f07cf5d0710e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:45:15.256242 9231 tablet_service.cc:1467] Processing CreateTablet for tablet 4a966fda8cd24551b2c7142fc0471c2a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3bbb5bcbf08544a8af28f07cf5d0710e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:45:15.256892 9371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a966fda8cd24551b2c7142fc0471c2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:15.256914 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a966fda8cd24551b2c7142fc0471c2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:15.256914 9231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a966fda8cd24551b2c7142fc0471c2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:15.259213 9600 tablet_bootstrap.cc:492] T 4a966fda8cd24551b2c7142fc0471c2a P 830f1c5539634e0b9645855927156696: Bootstrap starting.
I20260928 06:45:15.259686 9601 tablet_bootstrap.cc:492] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411: Bootstrap starting.
I20260928 06:45:15.260200 9602 tablet_bootstrap.cc:492] T 4a966fda8cd24551b2c7142fc0471c2a P e3f506d848e746fa8c27d0e3c46b8379: Bootstrap starting.
I20260928 06:45:15.260239 9600 tablet_bootstrap.cc:654] T 4a966fda8cd24551b2c7142fc0471c2a P 830f1c5539634e0b9645855927156696: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:15.260646 9601 tablet_bootstrap.cc:654] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:15.260996 9602 tablet_bootstrap.cc:654] T 4a966fda8cd24551b2c7142fc0471c2a P e3f506d848e746fa8c27d0e3c46b8379: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:15.261467 9600 tablet_bootstrap.cc:492] T 4a966fda8cd24551b2c7142fc0471c2a P 830f1c5539634e0b9645855927156696: No bootstrap required, opened a new log
I20260928 06:45:15.261579 9600 ts_tablet_manager.cc:1397] T 4a966fda8cd24551b2c7142fc0471c2a P 830f1c5539634e0b9645855927156696: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:15.262056 9601 tablet_bootstrap.cc:492] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411: No bootstrap required, opened a new log
I20260928 06:45:15.262146 9601 ts_tablet_manager.cc:1397] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:45:15.262365 9600 raft_consensus.cc:348] T 4a966fda8cd24551b2c7142fc0471c2a P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:15.262529 9600 raft_consensus.cc:374] T 4a966fda8cd24551b2c7142fc0471c2a P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:15.262567 9600 raft_consensus.cc:729] T 4a966fda8cd24551b2c7142fc0471c2a P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830f1c5539634e0b9645855927156696, State: Initialized, Role: FOLLOWER
I20260928 06:45:15.262611 9601 raft_consensus.cc:348] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:15.262701 9601 raft_consensus.cc:374] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:15.262733 9601 raft_consensus.cc:729] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 387ce6f9361a439f95ede8adcce0c411, State: Initialized, Role: FOLLOWER
I20260928 06:45:15.262697 9600 consensus_queue.cc:260] T 4a966fda8cd24551b2c7142fc0471c2a P 830f1c5539634e0b9645855927156696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:15.262866 9602 tablet_bootstrap.cc:492] T 4a966fda8cd24551b2c7142fc0471c2a P e3f506d848e746fa8c27d0e3c46b8379: No bootstrap required, opened a new log
I20260928 06:45:15.262934 9602 ts_tablet_manager.cc:1397] T 4a966fda8cd24551b2c7142fc0471c2a P e3f506d848e746fa8c27d0e3c46b8379: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:45:15.262882 9601 consensus_queue.cc:260] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:15.262971 9600 ts_tablet_manager.cc:1428] T 4a966fda8cd24551b2c7142fc0471c2a P 830f1c5539634e0b9645855927156696: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:45:15.263242 9601 ts_tablet_manager.cc:1428] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 06:45:15.263368 9602 raft_consensus.cc:348] T 4a966fda8cd24551b2c7142fc0471c2a P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:15.263482 9602 raft_consensus.cc:374] T 4a966fda8cd24551b2c7142fc0471c2a P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:15.263437 9606 tablet_replica.cc:439] T 4a966fda8cd24551b2c7142fc0471c2a P 830f1c5539634e0b9645855927156696: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } } }
I20260928 06:45:15.263532 9602 raft_consensus.cc:729] T 4a966fda8cd24551b2c7142fc0471c2a P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f506d848e746fa8c27d0e3c46b8379, State: Initialized, Role: FOLLOWER
I20260928 06:45:15.263597 9606 tablet_replica.cc:442] T 4a966fda8cd24551b2c7142fc0471c2a P 830f1c5539634e0b9645855927156696: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:45:15.263764 9607 raft_consensus.cc:482] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:45:15.263854 9607 raft_consensus.cc:504] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:15.264170 9602 consensus_queue.cc:260] T 4a966fda8cd24551b2c7142fc0471c2a P e3f506d848e746fa8c27d0e3c46b8379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:15.264426 9608 tablet_replica.cc:439] T 4a966fda8cd24551b2c7142fc0471c2a P e3f506d848e746fa8c27d0e3c46b8379: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } } }
I20260928 06:45:15.264544 9608 tablet_replica.cc:442] T 4a966fda8cd24551b2c7142fc0471c2a P e3f506d848e746fa8c27d0e3c46b8379: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:45:15.265110 9607 leader_election.cc:288] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3f506d848e746fa8c27d0e3c46b8379 (127.7.48.1:46737), 830f1c5539634e0b9645855927156696 (127.7.48.2:32827)
I20260928 06:45:15.265211 9602 ts_tablet_manager.cc:1428] T 4a966fda8cd24551b2c7142fc0471c2a P e3f506d848e746fa8c27d0e3c46b8379: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:45:15.265265 9607 tablet_replica.cc:439] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } } }
I20260928 06:45:15.265383 9607 tablet_replica.cc:442] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:45:15.270741 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:15.266311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:59411 (local address 127.7.48.2:32827)
0928 06:45:15.266419 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:15.266422 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:15.267076 (+ 654us) server_negotiation.cc:407] Connection header received
0928 06:45:15.267247 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:15.267250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:15.267294 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:15.267343 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:15.267748 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.268357 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.269518 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.270063 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.270159 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:15.270224 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:15.270568 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 06:45:15.270632 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:45:15.270773 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:15.267255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:48973 (local address 127.7.48.1:46737)
0928 06:45:15.267395 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:15.267399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:15.267416 (+ 17us) server_negotiation.cc:407] Connection header received
0928 06:45:15.267471 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:15.267474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:15.267542 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:15.267618 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:15.267994 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.268695 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.269680 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.270227 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.270325 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:15.270377 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:15.270578 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 06:45:15.270641 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64,"spinlock_wait_cycles":7424}
I20260928 06:45:15.270771 9470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:15.265690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:46737 (local address 127.7.48.3:48973)
0928 06:45:15.265831 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:15.265847 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:15.265921 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:15.267733 (+ 1812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:15.267738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:15.267753 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:15.267851 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:15.267858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.268800 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.268806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:15.269547 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:15.269557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.270339 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.270342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:15.270451 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:15.270470 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 06:45:15.270564 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 06:45:15.270630 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 06:45:15.270777 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:15.265921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:32827 (local address 127.7.48.3:59411)
0928 06:45:15.266986 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:15.267001 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:15.267139 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:15.267480 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:15.267483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:15.267499 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:15.267619 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:15.267626 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.268497 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.268503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:15.269392 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:15.269401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.270307 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.270310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:15.270426 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:15.270458 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 06:45:15.270552 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 06:45:15.270621 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"thread_start_us":88,"threads_started":1}
I20260928 06:45:15.271193 9391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a966fda8cd24551b2c7142fc0471c2a" candidate_uuid: "387ce6f9361a439f95ede8adcce0c411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830f1c5539634e0b9645855927156696" is_pre_election: true
I20260928 06:45:15.271322 9391 raft_consensus.cc:2454] T 4a966fda8cd24551b2c7142fc0471c2a P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 387ce6f9361a439f95ede8adcce0c411 in term 0.
I20260928 06:45:15.271430 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a966fda8cd24551b2c7142fc0471c2a" candidate_uuid: "387ce6f9361a439f95ede8adcce0c411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3f506d848e746fa8c27d0e3c46b8379" is_pre_election: true
I20260928 06:45:15.271673 9251 raft_consensus.cc:2454] T 4a966fda8cd24551b2c7142fc0471c2a P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 387ce6f9361a439f95ede8adcce0c411 in term 0.
I20260928 06:45:15.271673 9464 leader_election.cc:302] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 387ce6f9361a439f95ede8adcce0c411, 830f1c5539634e0b9645855927156696; no voters:
I20260928 06:45:15.271999 9622 raft_consensus.cc:2781] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:45:15.272096 9622 raft_consensus.cc:482] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:45:15.272152 9622 raft_consensus.cc:3037] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:15.273046 9622 raft_consensus.cc:504] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:15.273396 9622 leader_election.cc:288] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411 [CANDIDATE]: Term 1 election: Requested vote from peers e3f506d848e746fa8c27d0e3c46b8379 (127.7.48.1:46737), 830f1c5539634e0b9645855927156696 (127.7.48.2:32827)
I20260928 06:45:15.273814 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a966fda8cd24551b2c7142fc0471c2a" candidate_uuid: "387ce6f9361a439f95ede8adcce0c411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3f506d848e746fa8c27d0e3c46b8379"
I20260928 06:45:15.273921 9251 raft_consensus.cc:3037] T 4a966fda8cd24551b2c7142fc0471c2a P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:15.274717 9391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a966fda8cd24551b2c7142fc0471c2a" candidate_uuid: "387ce6f9361a439f95ede8adcce0c411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830f1c5539634e0b9645855927156696"
I20260928 06:45:15.274798 9391 raft_consensus.cc:3037] T 4a966fda8cd24551b2c7142fc0471c2a P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:15.274979 9251 raft_consensus.cc:2454] T 4a966fda8cd24551b2c7142fc0471c2a P e3f506d848e746fa8c27d0e3c46b8379 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 387ce6f9361a439f95ede8adcce0c411 in term 1.
I20260928 06:45:15.275341 9464 leader_election.cc:302] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 387ce6f9361a439f95ede8adcce0c411, e3f506d848e746fa8c27d0e3c46b8379; no voters:
I20260928 06:45:15.275496 9622 raft_consensus.cc:2781] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:15.275585 9622 raft_consensus.cc:686] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411 [term 1 LEADER]: Becoming Leader. State: Replica: 387ce6f9361a439f95ede8adcce0c411, State: Running, Role: LEADER
I20260928 06:45:15.275907 9391 raft_consensus.cc:2454] T 4a966fda8cd24551b2c7142fc0471c2a P 830f1c5539634e0b9645855927156696 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 387ce6f9361a439f95ede8adcce0c411 in term 1.
I20260928 06:45:15.275959 9622 consensus_queue.cc:237] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:15.277983 9607 tablet_replica.cc:439] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411: TxnStatusTablet state changed. Reason: New leader 387ce6f9361a439f95ede8adcce0c411. Latest consensus state: current_term: 1 leader_uuid: "387ce6f9361a439f95ede8adcce0c411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } } }
I20260928 06:45:15.278082 9607 tablet_replica.cc:442] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:45:15.278366 9625 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 06:45:15.278571 9625 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 06:45:15.279109 9093 catalog_manager.cc:5495] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411 reported cstate change: term changed from 0 to 1, leader changed from <none> to 387ce6f9361a439f95ede8adcce0c411 (127.7.48.3). New cstate: current_term: 1 leader_uuid: "387ce6f9361a439f95ede8adcce0c411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } health_report { overall_health: HEALTHY } } }
I20260928 06:45:15.293968 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:15.290157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:34929 (local address 127.0.0.1:36572)
0928 06:45:15.290307 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:15.290327 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:15.290412 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:15.290914 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:15.290917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:15.290925 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:45:15.291048 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:15.291053 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.291762 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.291767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:15.292422 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:15.292431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.292887 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.292891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:15.292986 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:15.293027 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:45:15.293682 (+ 655us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:45:15.293698 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 06:45:15.293767 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:45:15.293851 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 06:45:15.293989 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:15.290542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36572 (local address 127.7.48.3:34929)
0928 06:45:15.290670 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:15.290673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:15.290684 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:45:15.290739 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:15.290741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:15.290784 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:15.290900 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:45:15.291164 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.291649 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.292544 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.292765 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.292885 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:15.293179 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:45:15.293578 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:45:15.293705 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:15.293786 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 06:45:15.293834 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 06:45:15.298759 9251 raft_consensus.cc:1261] T 4a966fda8cd24551b2c7142fc0471c2a P e3f506d848e746fa8c27d0e3c46b8379 [term 1 FOLLOWER]: Refusing update from remote peer 387ce6f9361a439f95ede8adcce0c411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:15.298861 9391 raft_consensus.cc:1261] T 4a966fda8cd24551b2c7142fc0471c2a P 830f1c5539634e0b9645855927156696 [term 1 FOLLOWER]: Refusing update from remote peer 387ce6f9361a439f95ede8adcce0c411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:15.299345 9622 consensus_queue.cc:1035] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:15.299504 9607 consensus_queue.cc:1035] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:15.303097 9606 tablet_replica.cc:439] T 4a966fda8cd24551b2c7142fc0471c2a P 830f1c5539634e0b9645855927156696: TxnStatusTablet state changed. Reason: New leader 387ce6f9361a439f95ede8adcce0c411. Latest consensus state: current_term: 1 leader_uuid: "387ce6f9361a439f95ede8adcce0c411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } } }
I20260928 06:45:15.303223 9606 tablet_replica.cc:442] T 4a966fda8cd24551b2c7142fc0471c2a P 830f1c5539634e0b9645855927156696: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:45:15.303399 9608 tablet_replica.cc:439] T 4a966fda8cd24551b2c7142fc0471c2a P e3f506d848e746fa8c27d0e3c46b8379: TxnStatusTablet state changed. Reason: New leader 387ce6f9361a439f95ede8adcce0c411. Latest consensus state: current_term: 1 leader_uuid: "387ce6f9361a439f95ede8adcce0c411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } } }
I20260928 06:45:15.303508 9608 tablet_replica.cc:442] T 4a966fda8cd24551b2c7142fc0471c2a P e3f506d848e746fa8c27d0e3c46b8379: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:45:15.304358 9607 tablet_replica.cc:439] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "387ce6f9361a439f95ede8adcce0c411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } } }
I20260928 06:45:15.304451 9607 tablet_replica.cc:442] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:45:15.305619 9608 tablet_replica.cc:439] T 4a966fda8cd24551b2c7142fc0471c2a P e3f506d848e746fa8c27d0e3c46b8379: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "387ce6f9361a439f95ede8adcce0c411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } } }
I20260928 06:45:15.305723 9608 tablet_replica.cc:442] T 4a966fda8cd24551b2c7142fc0471c2a P e3f506d848e746fa8c27d0e3c46b8379: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:45:15.306437 9628 mvcc.cc:205] Tried to move back new op lower bound from 7334207141056917504 to 7334207140972744704. Current Snapshot: MvccSnapshot[applied={T|T < 7334207141056917504}]
I20260928 06:45:15.308207 9622 tablet_replica.cc:439] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "387ce6f9361a439f95ede8adcce0c411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } } }
I20260928 06:45:15.308300 9622 tablet_replica.cc:442] T 4a966fda8cd24551b2c7142fc0471c2a P 387ce6f9361a439f95ede8adcce0c411: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:45:15.309370 9606 tablet_replica.cc:439] T 4a966fda8cd24551b2c7142fc0471c2a P 830f1c5539634e0b9645855927156696: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "387ce6f9361a439f95ede8adcce0c411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } } }
I20260928 06:45:15.309487 9606 tablet_replica.cc:442] T 4a966fda8cd24551b2c7142fc0471c2a P 830f1c5539634e0b9645855927156696: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:45:15.309559 9629 mvcc.cc:205] Tried to move back new op lower bound from 7334207141056917504 to 7334207140972744704. Current Snapshot: MvccSnapshot[applied={T|T < 7334207141056917504}]
I20260928 06:45:15.310361 9630 mvcc.cc:205] Tried to move back new op lower bound from 7334207141056917504 to 7334207140972744704. Current Snapshot: MvccSnapshot[applied={T|T < 7334207141056917504}]
I20260928 06:45:15.319548 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:15.315925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59024 (local address 127.7.48.1:46737)
0928 06:45:15.316045 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:15.316049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:15.316064 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:45:15.316135 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:15.316138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:15.316193 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:15.316296 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:45:15.316516 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.317329 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.318128 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.318478 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.318593 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:15.318804 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:45:15.319231 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:45:15.319333 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:15.319370 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 06:45:15.319422 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:45:15.329455 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:15.326242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:57773 (local address 127.7.48.3:34929)
0928 06:45:15.326353 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:15.326357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:15.326559 (+ 202us) server_negotiation.cc:407] Connection header received
0928 06:45:15.326728 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:15.326730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:15.326766 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:15.326840 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:45:15.327104 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.327571 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.328484 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.328685 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.328784 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:15.329050 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:45:15.329113 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:45:15.329184 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:15.329267 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 06:45:15.329309 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:45:15.329521 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:15.326155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:34929 (local address 127.7.48.1:57773)
0928 06:45:15.326453 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:15.326464 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:15.326599 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:15.326861 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:15.326864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:15.326871 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:45:15.326996 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:15.327001 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.327682 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.327688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:15.328371 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:15.328379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.328785 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.328788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:15.328870 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:15.328913 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:45:15.329189 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:45:15.329197 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 06:45:15.329257 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:45:15.329323 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":92,"threads_started":1}
I20260928 06:45:15.336237 9391 raft_consensus.cc:1261] T de7a9595979546b8a0cda9be393b69f6 P 830f1c5539634e0b9645855927156696 [term 1 FOLLOWER]: Refusing update from remote peer e3f506d848e746fa8c27d0e3c46b8379: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:15.336678 9611 consensus_queue.cc:1035] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379 [LEADER]: Connected to new peer: Peer: permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:15.336789 9531 raft_consensus.cc:1261] T de7a9595979546b8a0cda9be393b69f6 P 387ce6f9361a439f95ede8adcce0c411 [term 1 FOLLOWER]: Refusing update from remote peer e3f506d848e746fa8c27d0e3c46b8379: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:15.337394 9611 consensus_queue.cc:1035] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379 [LEADER]: Connected to new peer: Peer: permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:15.357919 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:15.354412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:38429 (local address 127.7.48.1:46737)
0928 06:45:15.354530 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:15.354535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:15.354547 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:45:15.354588 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:15.354591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:15.354633 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:15.354718 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:45:15.355013 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.355689 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.356814 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.357015 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.357159 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:15.357413 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:45:15.357509 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:45:15.357583 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:15.357744 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 06:45:15.357796 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:45:15.357915 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:15.354051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:46737 (local address 127.7.48.3:38429)
0928 06:45:15.354209 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:15.354227 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:15.354321 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:15.354742 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:15.354745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:15.354753 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:45:15.354866 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:15.354872 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.355815 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.355818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:15.356689 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:15.356701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:15.357149 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:15.357152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:15.357248 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:15.357297 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:45:15.357612 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:45:15.357619 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 06:45:15.357708 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:45:15.357781 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 06:45:15.380317 9092 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:15.380487 9092 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:15.382714 9092 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7: Sending DeleteTablet for 3 replicas of tablet de7a9595979546b8a0cda9be393b69f6
I20260928 06:45:15.383471 9511 tablet_service.cc:1514] Processing DeleteTablet for tablet de7a9595979546b8a0cda9be393b69f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:15 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:36558
I20260928 06:45:15.383474 9231 tablet_service.cc:1514] Processing DeleteTablet for tablet de7a9595979546b8a0cda9be393b69f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:15 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:59014
I20260928 06:45:15.383857 9648 tablet_replica.cc:330] T de7a9595979546b8a0cda9be393b69f6 P 387ce6f9361a439f95ede8adcce0c411: stopping tablet replica
I20260928 06:45:15.384048 9371 tablet_service.cc:1514] Processing DeleteTablet for tablet de7a9595979546b8a0cda9be393b69f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:15 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:43950
I20260928 06:45:15.384284 9648 raft_consensus.cc:2229] T de7a9595979546b8a0cda9be393b69f6 P 387ce6f9361a439f95ede8adcce0c411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:15.384497 9648 raft_consensus.cc:2258] T de7a9595979546b8a0cda9be393b69f6 P 387ce6f9361a439f95ede8adcce0c411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:15.385094 9092 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:45:15.385512 9092 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:45:15.386106 9648 ts_tablet_manager.cc:1905] T de7a9595979546b8a0cda9be393b69f6 P 387ce6f9361a439f95ede8adcce0c411: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:15.389292 9648 ts_tablet_manager.cc:1918] T de7a9595979546b8a0cda9be393b69f6 P 387ce6f9361a439f95ede8adcce0c411: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:45:15.389468 9648 log.cc:1196] T de7a9595979546b8a0cda9be393b69f6 P 387ce6f9361a439f95ede8adcce0c411: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/de7a9595979546b8a0cda9be393b69f6
I20260928 06:45:15.389875 9648 ts_tablet_manager.cc:1939] T de7a9595979546b8a0cda9be393b69f6 P 387ce6f9361a439f95ede8adcce0c411: Deleting consensus metadata
I20260928 06:45:15.391184 9231 tablet_service.cc:1467] Processing CreateTablet for tablet cdcc538650a6425a965d64b5ee16e701 (DEFAULT_TABLE table=test-table [id=69dbf64e5290439b929f978c5187b24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:15.391462 9231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdcc538650a6425a965d64b5ee16e701. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:15.392848 9650 tablet_replica.cc:330] T de7a9595979546b8a0cda9be393b69f6 P 830f1c5539634e0b9645855927156696: stopping tablet replica
I20260928 06:45:15.393230 9650 raft_consensus.cc:2229] T de7a9595979546b8a0cda9be393b69f6 P 830f1c5539634e0b9645855927156696 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:15.393537 9650 raft_consensus.cc:2258] T de7a9595979546b8a0cda9be393b69f6 P 830f1c5539634e0b9645855927156696 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:15.394102 9602 tablet_bootstrap.cc:492] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379: Bootstrap starting.
I20260928 06:45:15.394095 9649 tablet_replica.cc:330] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379: stopping tablet replica
I20260928 06:45:15.394411 9649 raft_consensus.cc:2229] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:45:15.394744 9649 raft_consensus.cc:2258] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:15.395069 9602 tablet_bootstrap.cc:654] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:15.395274 9650 ts_tablet_manager.cc:1905] T de7a9595979546b8a0cda9be393b69f6 P 830f1c5539634e0b9645855927156696: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:15.396050 9649 ts_tablet_manager.cc:1905] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:15.396116 9371 tablet_service.cc:1467] Processing CreateTablet for tablet cdcc538650a6425a965d64b5ee16e701 (DEFAULT_TABLE table=test-table [id=69dbf64e5290439b929f978c5187b24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:15.396377 9371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdcc538650a6425a965d64b5ee16e701. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:15.398447 9650 ts_tablet_manager.cc:1918] T de7a9595979546b8a0cda9be393b69f6 P 830f1c5539634e0b9645855927156696: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:45:15.398588 9650 log.cc:1196] T de7a9595979546b8a0cda9be393b69f6 P 830f1c5539634e0b9645855927156696: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/de7a9595979546b8a0cda9be393b69f6
I20260928 06:45:15.398607 9600 tablet_bootstrap.cc:492] T cdcc538650a6425a965d64b5ee16e701 P 830f1c5539634e0b9645855927156696: Bootstrap starting.
I20260928 06:45:15.398730 9649 ts_tablet_manager.cc:1918] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:45:15.398826 9649 log.cc:1196] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-0/wal/wals/de7a9595979546b8a0cda9be393b69f6
I20260928 06:45:15.399166 9649 ts_tablet_manager.cc:1939] T de7a9595979546b8a0cda9be393b69f6 P e3f506d848e746fa8c27d0e3c46b8379: Deleting consensus metadata
I20260928 06:45:15.399195 9650 ts_tablet_manager.cc:1939] T de7a9595979546b8a0cda9be393b69f6 P 830f1c5539634e0b9645855927156696: Deleting consensus metadata
I20260928 06:45:15.399736 9600 tablet_bootstrap.cc:654] T cdcc538650a6425a965d64b5ee16e701 P 830f1c5539634e0b9645855927156696: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:15.400547 9511 tablet_service.cc:1467] Processing CreateTablet for tablet cdcc538650a6425a965d64b5ee16e701 (DEFAULT_TABLE table=test-table [id=69dbf64e5290439b929f978c5187b24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:15.400822 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdcc538650a6425a965d64b5ee16e701. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:15.402529 9601 tablet_bootstrap.cc:492] T cdcc538650a6425a965d64b5ee16e701 P 387ce6f9361a439f95ede8adcce0c411: Bootstrap starting.
I20260928 06:45:15.402876 9080 catalog_manager.cc:4841] TS 830f1c5539634e0b9645855927156696 (127.7.48.2:32827): tablet de7a9595979546b8a0cda9be393b69f6 (table test-table [id=dd0aae8553db4f6986a45e235d3a0803]) successfully deleted
I20260928 06:45:15.403090 9080 catalog_manager.cc:4841] TS e3f506d848e746fa8c27d0e3c46b8379 (127.7.48.1:46737): tablet de7a9595979546b8a0cda9be393b69f6 (table test-table [id=dd0aae8553db4f6986a45e235d3a0803]) successfully deleted
I20260928 06:45:15.403203 9080 catalog_manager.cc:4841] TS 387ce6f9361a439f95ede8adcce0c411 (127.7.48.3:34929): tablet de7a9595979546b8a0cda9be393b69f6 (table test-table [id=dd0aae8553db4f6986a45e235d3a0803]) successfully deleted
I20260928 06:45:15.403324 9601 tablet_bootstrap.cc:654] T cdcc538650a6425a965d64b5ee16e701 P 387ce6f9361a439f95ede8adcce0c411: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:15.404284 9602 tablet_bootstrap.cc:492] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379: No bootstrap required, opened a new log
I20260928 06:45:15.404366 9602 ts_tablet_manager.cc:1397] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260928 06:45:15.404523 9601 tablet_bootstrap.cc:492] T cdcc538650a6425a965d64b5ee16e701 P 387ce6f9361a439f95ede8adcce0c411: No bootstrap required, opened a new log
I20260928 06:45:15.404584 9601 ts_tablet_manager.cc:1397] T cdcc538650a6425a965d64b5ee16e701 P 387ce6f9361a439f95ede8adcce0c411: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:45:15.404799 9602 raft_consensus.cc:348] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:15.404924 9602 raft_consensus.cc:374] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:15.404963 9602 raft_consensus.cc:729] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f506d848e746fa8c27d0e3c46b8379, State: Initialized, Role: FOLLOWER
I20260928 06:45:15.404964 9601 raft_consensus.cc:348] T cdcc538650a6425a965d64b5ee16e701 P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:15.405063 9601 raft_consensus.cc:374] T cdcc538650a6425a965d64b5ee16e701 P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:15.405093 9601 raft_consensus.cc:729] T cdcc538650a6425a965d64b5ee16e701 P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 387ce6f9361a439f95ede8adcce0c411, State: Initialized, Role: FOLLOWER
I20260928 06:45:15.405085 9602 consensus_queue.cc:260] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:15.405211 9601 consensus_queue.cc:260] T cdcc538650a6425a965d64b5ee16e701 P 387ce6f9361a439f95ede8adcce0c411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:15.405381 9602 ts_tablet_manager.cc:1428] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:45:15.405460 9601 ts_tablet_manager.cc:1428] T cdcc538650a6425a965d64b5ee16e701 P 387ce6f9361a439f95ede8adcce0c411: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:45:15.406607 9600 tablet_bootstrap.cc:492] T cdcc538650a6425a965d64b5ee16e701 P 830f1c5539634e0b9645855927156696: No bootstrap required, opened a new log
I20260928 06:45:15.406689 9600 ts_tablet_manager.cc:1397] T cdcc538650a6425a965d64b5ee16e701 P 830f1c5539634e0b9645855927156696: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260928 06:45:15.407079 9600 raft_consensus.cc:348] T cdcc538650a6425a965d64b5ee16e701 P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:15.407191 9600 raft_consensus.cc:374] T cdcc538650a6425a965d64b5ee16e701 P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:15.407218 9600 raft_consensus.cc:729] T cdcc538650a6425a965d64b5ee16e701 P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830f1c5539634e0b9645855927156696, State: Initialized, Role: FOLLOWER
I20260928 06:45:15.407330 9600 consensus_queue.cc:260] T cdcc538650a6425a965d64b5ee16e701 P 830f1c5539634e0b9645855927156696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:15.407586 9600 ts_tablet_manager.cc:1428] T cdcc538650a6425a965d64b5ee16e701 P 830f1c5539634e0b9645855927156696: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:45:15.765828 9644 raft_consensus.cc:482] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:45:15.766014 9644 raft_consensus.cc:504] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:15.766430 9644 leader_election.cc:288] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 830f1c5539634e0b9645855927156696 (127.7.48.2:32827), 387ce6f9361a439f95ede8adcce0c411 (127.7.48.3:34929)
I20260928 06:45:15.766959 9391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdcc538650a6425a965d64b5ee16e701" candidate_uuid: "e3f506d848e746fa8c27d0e3c46b8379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830f1c5539634e0b9645855927156696" is_pre_election: true
I20260928 06:45:15.766959 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdcc538650a6425a965d64b5ee16e701" candidate_uuid: "e3f506d848e746fa8c27d0e3c46b8379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "387ce6f9361a439f95ede8adcce0c411" is_pre_election: true
I20260928 06:45:15.767112 9531 raft_consensus.cc:2454] T cdcc538650a6425a965d64b5ee16e701 P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3f506d848e746fa8c27d0e3c46b8379 in term 0.
I20260928 06:45:15.767112 9391 raft_consensus.cc:2454] T cdcc538650a6425a965d64b5ee16e701 P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3f506d848e746fa8c27d0e3c46b8379 in term 0.
I20260928 06:45:15.767418 9184 leader_election.cc:302] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 387ce6f9361a439f95ede8adcce0c411, e3f506d848e746fa8c27d0e3c46b8379; no voters:
I20260928 06:45:15.767621 9644 raft_consensus.cc:2781] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:45:15.767699 9644 raft_consensus.cc:482] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:45:15.767752 9644 raft_consensus.cc:3037] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:15.768829 9644 raft_consensus.cc:504] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:15.769152 9644 leader_election.cc:288] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379 [CANDIDATE]: Term 1 election: Requested vote from peers 830f1c5539634e0b9645855927156696 (127.7.48.2:32827), 387ce6f9361a439f95ede8adcce0c411 (127.7.48.3:34929)
I20260928 06:45:15.769528 9391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdcc538650a6425a965d64b5ee16e701" candidate_uuid: "e3f506d848e746fa8c27d0e3c46b8379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830f1c5539634e0b9645855927156696"
I20260928 06:45:15.769598 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdcc538650a6425a965d64b5ee16e701" candidate_uuid: "e3f506d848e746fa8c27d0e3c46b8379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "387ce6f9361a439f95ede8adcce0c411"
I20260928 06:45:15.769680 9391 raft_consensus.cc:3037] T cdcc538650a6425a965d64b5ee16e701 P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:15.769765 9531 raft_consensus.cc:3037] T cdcc538650a6425a965d64b5ee16e701 P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:15.770653 9391 raft_consensus.cc:2454] T cdcc538650a6425a965d64b5ee16e701 P 830f1c5539634e0b9645855927156696 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3f506d848e746fa8c27d0e3c46b8379 in term 1.
I20260928 06:45:15.770787 9531 raft_consensus.cc:2454] T cdcc538650a6425a965d64b5ee16e701 P 387ce6f9361a439f95ede8adcce0c411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3f506d848e746fa8c27d0e3c46b8379 in term 1.
I20260928 06:45:15.771174 9184 leader_election.cc:302] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 387ce6f9361a439f95ede8adcce0c411, e3f506d848e746fa8c27d0e3c46b8379; no voters:
I20260928 06:45:15.771394 9644 raft_consensus.cc:2781] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:15.771505 9644 raft_consensus.cc:686] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379 [term 1 LEADER]: Becoming Leader. State: Replica: e3f506d848e746fa8c27d0e3c46b8379, State: Running, Role: LEADER
I20260928 06:45:15.771653 9644 consensus_queue.cc:237] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:15.773418 9092 catalog_manager.cc:5495] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3f506d848e746fa8c27d0e3c46b8379 (127.7.48.1). New cstate: current_term: 1 leader_uuid: "e3f506d848e746fa8c27d0e3c46b8379" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } health_report { overall_health: UNKNOWN } } }
I20260928 06:45:15.868628 9391 raft_consensus.cc:1261] T cdcc538650a6425a965d64b5ee16e701 P 830f1c5539634e0b9645855927156696 [term 1 FOLLOWER]: Refusing update from remote peer e3f506d848e746fa8c27d0e3c46b8379: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:15.868705 9531 raft_consensus.cc:1261] T cdcc538650a6425a965d64b5ee16e701 P 387ce6f9361a439f95ede8adcce0c411 [term 1 FOLLOWER]: Refusing update from remote peer e3f506d848e746fa8c27d0e3c46b8379: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:15.869118 9646 consensus_queue.cc:1035] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379 [LEADER]: Connected to new peer: Peer: permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:15.869287 9644 consensus_queue.cc:1035] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379 [LEADER]: Connected to new peer: Peer: permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 06:45:15.882745 9630 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 06:45:17.433331 9092 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:17.433495 9092 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:17.435824 9092 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7: Sending DeleteTablet for 3 replicas of tablet cdcc538650a6425a965d64b5ee16e701
I20260928 06:45:17.436420 9231 tablet_service.cc:1514] Processing DeleteTablet for tablet cdcc538650a6425a965d64b5ee16e701 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:17 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:59014
I20260928 06:45:17.436522 9371 tablet_service.cc:1514] Processing DeleteTablet for tablet cdcc538650a6425a965d64b5ee16e701 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:17 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:43950
I20260928 06:45:17.436532 9511 tablet_service.cc:1514] Processing DeleteTablet for tablet cdcc538650a6425a965d64b5ee16e701 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:17 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:36558
I20260928 06:45:17.436795 9682 tablet_replica.cc:330] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379: stopping tablet replica
I20260928 06:45:17.436866 9683 tablet_replica.cc:330] T cdcc538650a6425a965d64b5ee16e701 P 830f1c5539634e0b9645855927156696: stopping tablet replica
I20260928 06:45:17.436941 9682 raft_consensus.cc:2229] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:45:17.437029 9683 raft_consensus.cc:2229] T cdcc538650a6425a965d64b5ee16e701 P 830f1c5539634e0b9645855927156696 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:17.437155 9682 raft_consensus.cc:2258] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:17.437170 9683 raft_consensus.cc:2258] T cdcc538650a6425a965d64b5ee16e701 P 830f1c5539634e0b9645855927156696 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:17.437418 9684 tablet_replica.cc:330] T cdcc538650a6425a965d64b5ee16e701 P 387ce6f9361a439f95ede8adcce0c411: stopping tablet replica
I20260928 06:45:17.437577 9684 raft_consensus.cc:2229] T cdcc538650a6425a965d64b5ee16e701 P 387ce6f9361a439f95ede8adcce0c411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:17.437703 9684 raft_consensus.cc:2258] T cdcc538650a6425a965d64b5ee16e701 P 387ce6f9361a439f95ede8adcce0c411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:17.437932 9682 ts_tablet_manager.cc:1905] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:17.438105 9683 ts_tablet_manager.cc:1905] T cdcc538650a6425a965d64b5ee16e701 P 830f1c5539634e0b9645855927156696: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:17.440536 9684 ts_tablet_manager.cc:1905] T cdcc538650a6425a965d64b5ee16e701 P 387ce6f9361a439f95ede8adcce0c411: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:17.441286 9683 ts_tablet_manager.cc:1918] T cdcc538650a6425a965d64b5ee16e701 P 830f1c5539634e0b9645855927156696: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:45:17.441442 9683 log.cc:1196] T cdcc538650a6425a965d64b5ee16e701 P 830f1c5539634e0b9645855927156696: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/cdcc538650a6425a965d64b5ee16e701
I20260928 06:45:17.441576 9682 ts_tablet_manager.cc:1918] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:45:17.441663 9682 log.cc:1196] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-0/wal/wals/cdcc538650a6425a965d64b5ee16e701
I20260928 06:45:17.441892 9683 ts_tablet_manager.cc:1939] T cdcc538650a6425a965d64b5ee16e701 P 830f1c5539634e0b9645855927156696: Deleting consensus metadata
I20260928 06:45:17.441977 9682 ts_tablet_manager.cc:1939] T cdcc538650a6425a965d64b5ee16e701 P e3f506d848e746fa8c27d0e3c46b8379: Deleting consensus metadata
I20260928 06:45:17.442698 9080 catalog_manager.cc:4841] TS 830f1c5539634e0b9645855927156696 (127.7.48.2:32827): tablet cdcc538650a6425a965d64b5ee16e701 (table test-table [id=69dbf64e5290439b929f978c5187b24e]) successfully deleted
I20260928 06:45:17.442862 9080 catalog_manager.cc:4841] TS e3f506d848e746fa8c27d0e3c46b8379 (127.7.48.1:46737): tablet cdcc538650a6425a965d64b5ee16e701 (table test-table [id=69dbf64e5290439b929f978c5187b24e]) successfully deleted
I20260928 06:45:17.443296 9684 ts_tablet_manager.cc:1918] T cdcc538650a6425a965d64b5ee16e701 P 387ce6f9361a439f95ede8adcce0c411: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:45:17.443389 9684 log.cc:1196] T cdcc538650a6425a965d64b5ee16e701 P 387ce6f9361a439f95ede8adcce0c411: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/cdcc538650a6425a965d64b5ee16e701
I20260928 06:45:17.443645 9684 ts_tablet_manager.cc:1939] T cdcc538650a6425a965d64b5ee16e701 P 387ce6f9361a439f95ede8adcce0c411: Deleting consensus metadata
I20260928 06:45:17.444326 9080 catalog_manager.cc:4841] TS 387ce6f9361a439f95ede8adcce0c411 (127.7.48.3:34929): tablet cdcc538650a6425a965d64b5ee16e701 (table test-table [id=69dbf64e5290439b929f978c5187b24e]) successfully deleted
Sep 28 06:45:17 dist-test-slave-twl6 krb5kdc[9049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577917, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:45:17 dist-test-slave-twl6 krb5kdc[9049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577917, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:17.471076 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:17.455786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59816 (local address 127.7.48.62:44081)
0928 06:45:17.456090 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:17.456097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:17.456112 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:45:17.456183 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:17.456185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:17.456226 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:17.456305 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:17.457054 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:17.457600 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:17.458461 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:17.458671 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:17.458744 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:17.467685 (+ 8941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:17.467688 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:17.467690 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:17.467716 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:17.469359 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:17.469774 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:17.469777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:17.469778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:17.469852 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:17.470141 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:17.470145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:17.470147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:17.470350 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:17.470523 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:17.470764 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 06:45:17.470892 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":132,"threads_started":1}
I20260928 06:45:17.473647 9092 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:45:17.474107 9092 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:45:17.478895 9231 tablet_service.cc:1467] Processing CreateTablet for tablet aad4288d6b1c49588682fd266b5e50fa (DEFAULT_TABLE table=test-table [id=dc3b722800b740f2b1a03ed6f56877de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:17.478904 9371 tablet_service.cc:1467] Processing CreateTablet for tablet aad4288d6b1c49588682fd266b5e50fa (DEFAULT_TABLE table=test-table [id=dc3b722800b740f2b1a03ed6f56877de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:17.478927 9511 tablet_service.cc:1467] Processing CreateTablet for tablet aad4288d6b1c49588682fd266b5e50fa (DEFAULT_TABLE table=test-table [id=dc3b722800b740f2b1a03ed6f56877de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:17.479197 9371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aad4288d6b1c49588682fd266b5e50fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:17.479197 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aad4288d6b1c49588682fd266b5e50fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:17.479210 9231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aad4288d6b1c49588682fd266b5e50fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:17.481386 9693 tablet_bootstrap.cc:492] T aad4288d6b1c49588682fd266b5e50fa P e3f506d848e746fa8c27d0e3c46b8379: Bootstrap starting.
I20260928 06:45:17.481678 9695 tablet_bootstrap.cc:492] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411: Bootstrap starting.
I20260928 06:45:17.481879 9694 tablet_bootstrap.cc:492] T aad4288d6b1c49588682fd266b5e50fa P 830f1c5539634e0b9645855927156696: Bootstrap starting.
I20260928 06:45:17.482355 9693 tablet_bootstrap.cc:654] T aad4288d6b1c49588682fd266b5e50fa P e3f506d848e746fa8c27d0e3c46b8379: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:17.482758 9695 tablet_bootstrap.cc:654] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:17.483148 9694 tablet_bootstrap.cc:654] T aad4288d6b1c49588682fd266b5e50fa P 830f1c5539634e0b9645855927156696: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:17.483656 9693 tablet_bootstrap.cc:492] T aad4288d6b1c49588682fd266b5e50fa P e3f506d848e746fa8c27d0e3c46b8379: No bootstrap required, opened a new log
I20260928 06:45:17.483752 9693 ts_tablet_manager.cc:1397] T aad4288d6b1c49588682fd266b5e50fa P e3f506d848e746fa8c27d0e3c46b8379: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:17.483976 9695 tablet_bootstrap.cc:492] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411: No bootstrap required, opened a new log
I20260928 06:45:17.484071 9695 ts_tablet_manager.cc:1397] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:17.484249 9693 raft_consensus.cc:348] T aad4288d6b1c49588682fd266b5e50fa P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:17.484373 9693 raft_consensus.cc:374] T aad4288d6b1c49588682fd266b5e50fa P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:17.484409 9693 raft_consensus.cc:729] T aad4288d6b1c49588682fd266b5e50fa P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f506d848e746fa8c27d0e3c46b8379, State: Initialized, Role: FOLLOWER
I20260928 06:45:17.484494 9694 tablet_bootstrap.cc:492] T aad4288d6b1c49588682fd266b5e50fa P 830f1c5539634e0b9645855927156696: No bootstrap required, opened a new log
I20260928 06:45:17.484506 9695 raft_consensus.cc:348] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:17.484575 9694 ts_tablet_manager.cc:1397] T aad4288d6b1c49588682fd266b5e50fa P 830f1c5539634e0b9645855927156696: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:45:17.484601 9695 raft_consensus.cc:374] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:17.484634 9695 raft_consensus.cc:729] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 387ce6f9361a439f95ede8adcce0c411, State: Initialized, Role: FOLLOWER
I20260928 06:45:17.484588 9693 consensus_queue.cc:260] T aad4288d6b1c49588682fd266b5e50fa P e3f506d848e746fa8c27d0e3c46b8379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:17.484956 9693 ts_tablet_manager.cc:1428] T aad4288d6b1c49588682fd266b5e50fa P e3f506d848e746fa8c27d0e3c46b8379: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:45:17.485035 9694 raft_consensus.cc:348] T aad4288d6b1c49588682fd266b5e50fa P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:17.485167 9694 raft_consensus.cc:374] T aad4288d6b1c49588682fd266b5e50fa P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:17.485229 9694 raft_consensus.cc:729] T aad4288d6b1c49588682fd266b5e50fa P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830f1c5539634e0b9645855927156696, State: Initialized, Role: FOLLOWER
I20260928 06:45:17.485304 9695 consensus_queue.cc:260] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:17.485397 9694 consensus_queue.cc:260] T aad4288d6b1c49588682fd266b5e50fa P 830f1c5539634e0b9645855927156696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:17.486097 9695 ts_tablet_manager.cc:1428] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:45:17.485863 9694 ts_tablet_manager.cc:1428] T aad4288d6b1c49588682fd266b5e50fa P 830f1c5539634e0b9645855927156696: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 06:45:17.503327 9663 raft_consensus.cc:482] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:45:17.503454 9663 raft_consensus.cc:504] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:17.503831 9663 leader_election.cc:288] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3f506d848e746fa8c27d0e3c46b8379 (127.7.48.1:46737), 830f1c5539634e0b9645855927156696 (127.7.48.2:32827)
I20260928 06:45:17.504213 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad4288d6b1c49588682fd266b5e50fa" candidate_uuid: "387ce6f9361a439f95ede8adcce0c411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3f506d848e746fa8c27d0e3c46b8379" is_pre_election: true
I20260928 06:45:17.504350 9251 raft_consensus.cc:2454] T aad4288d6b1c49588682fd266b5e50fa P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 387ce6f9361a439f95ede8adcce0c411 in term 0.
I20260928 06:45:17.504359 9391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad4288d6b1c49588682fd266b5e50fa" candidate_uuid: "387ce6f9361a439f95ede8adcce0c411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830f1c5539634e0b9645855927156696" is_pre_election: true
I20260928 06:45:17.504550 9391 raft_consensus.cc:2454] T aad4288d6b1c49588682fd266b5e50fa P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 387ce6f9361a439f95ede8adcce0c411 in term 0.
I20260928 06:45:17.504647 9464 leader_election.cc:302] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 387ce6f9361a439f95ede8adcce0c411, e3f506d848e746fa8c27d0e3c46b8379; no voters:
I20260928 06:45:17.504825 9663 raft_consensus.cc:2781] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:45:17.504892 9663 raft_consensus.cc:482] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:45:17.504943 9663 raft_consensus.cc:3037] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:17.506145 9663 raft_consensus.cc:504] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:17.506471 9663 leader_election.cc:288] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411 [CANDIDATE]: Term 1 election: Requested vote from peers e3f506d848e746fa8c27d0e3c46b8379 (127.7.48.1:46737), 830f1c5539634e0b9645855927156696 (127.7.48.2:32827)
I20260928 06:45:17.506781 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad4288d6b1c49588682fd266b5e50fa" candidate_uuid: "387ce6f9361a439f95ede8adcce0c411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3f506d848e746fa8c27d0e3c46b8379"
I20260928 06:45:17.506876 9251 raft_consensus.cc:3037] T aad4288d6b1c49588682fd266b5e50fa P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:17.506889 9391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad4288d6b1c49588682fd266b5e50fa" candidate_uuid: "387ce6f9361a439f95ede8adcce0c411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830f1c5539634e0b9645855927156696"
I20260928 06:45:17.507009 9391 raft_consensus.cc:3037] T aad4288d6b1c49588682fd266b5e50fa P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:17.507822 9251 raft_consensus.cc:2454] T aad4288d6b1c49588682fd266b5e50fa P e3f506d848e746fa8c27d0e3c46b8379 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 387ce6f9361a439f95ede8adcce0c411 in term 1.
I20260928 06:45:17.507864 9391 raft_consensus.cc:2454] T aad4288d6b1c49588682fd266b5e50fa P 830f1c5539634e0b9645855927156696 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 387ce6f9361a439f95ede8adcce0c411 in term 1.
I20260928 06:45:17.508149 9464 leader_election.cc:302] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 387ce6f9361a439f95ede8adcce0c411, e3f506d848e746fa8c27d0e3c46b8379; no voters:
I20260928 06:45:17.508364 9663 raft_consensus.cc:2781] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:17.508525 9663 raft_consensus.cc:686] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411 [term 1 LEADER]: Becoming Leader. State: Replica: 387ce6f9361a439f95ede8adcce0c411, State: Running, Role: LEADER
I20260928 06:45:17.508673 9663 consensus_queue.cc:237] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:17.510232 9092 catalog_manager.cc:5495] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411 reported cstate change: term changed from 0 to 1, leader changed from <none> to 387ce6f9361a439f95ede8adcce0c411 (127.7.48.3). New cstate: current_term: 1 leader_uuid: "387ce6f9361a439f95ede8adcce0c411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } health_report { overall_health: HEALTHY } } }
I20260928 06:45:17.524405 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:17.521415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36578 (local address 127.7.48.3:34929)
0928 06:45:17.521662 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:17.521668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:17.521680 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:45:17.521717 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:17.521718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:17.521756 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:17.521830 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:45:17.522078 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:17.522612 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:17.523393 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:17.523616 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:17.523690 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:17.523957 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:45:17.524027 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:45:17.524083 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:17.524168 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 06:45:17.524211 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":110,"threads_started":1}
I20260928 06:45:17.533111 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:17.530260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:36307 (local address 127.7.48.3:34929)
0928 06:45:17.530379 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:17.530383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:17.530485 (+ 102us) server_negotiation.cc:407] Connection header received
0928 06:45:17.530608 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:17.530610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:17.530656 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:17.530719 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:45:17.530927 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:17.531402 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:17.532281 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:17.532482 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:17.532558 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:17.532775 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:45:17.532834 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:45:17.532886 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:17.532967 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 06:45:17.533007 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:45:17.533131 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:17.530150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:34929 (local address 127.7.48.3:36307)
0928 06:45:17.530402 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:17.530414 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:17.530523 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:17.530746 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:17.530748 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:17.530755 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:45:17.530840 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:17.530844 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:17.531509 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:17.531512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:17.532179 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:17.532188 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:17.532585 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:17.532587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:17.532650 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:17.532675 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:45:17.532908 (+ 233us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:45:17.532912 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:45:17.532959 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 06:45:17.533022 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":1,"thread_start_us":105,"threads_started":1}
I20260928 06:45:17.538985 9391 raft_consensus.cc:1261] T aad4288d6b1c49588682fd266b5e50fa P 830f1c5539634e0b9645855927156696 [term 1 FOLLOWER]: Refusing update from remote peer 387ce6f9361a439f95ede8adcce0c411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:17.539050 9251 raft_consensus.cc:1261] T aad4288d6b1c49588682fd266b5e50fa P e3f506d848e746fa8c27d0e3c46b8379 [term 1 FOLLOWER]: Refusing update from remote peer 387ce6f9361a439f95ede8adcce0c411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:17.539433 9660 consensus_queue.cc:1035] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:17.539619 9668 consensus_queue.cc:1035] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:17.573907 9092 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:17.574086 9092 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:17.576251 9092 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7: Sending DeleteTablet for 3 replicas of tablet aad4288d6b1c49588682fd266b5e50fa
I20260928 06:45:17.576856 9231 tablet_service.cc:1514] Processing DeleteTablet for tablet aad4288d6b1c49588682fd266b5e50fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:17 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:59014
I20260928 06:45:17.576918 9371 tablet_service.cc:1514] Processing DeleteTablet for tablet aad4288d6b1c49588682fd266b5e50fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:17 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:43950
I20260928 06:45:17.576978 9510 tablet_service.cc:1514] Processing DeleteTablet for tablet aad4288d6b1c49588682fd266b5e50fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:17 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:36558
I20260928 06:45:17.577045 9682 tablet_replica.cc:330] T aad4288d6b1c49588682fd266b5e50fa P e3f506d848e746fa8c27d0e3c46b8379: stopping tablet replica
I20260928 06:45:17.577114 9683 tablet_replica.cc:330] T aad4288d6b1c49588682fd266b5e50fa P 830f1c5539634e0b9645855927156696: stopping tablet replica
I20260928 06:45:17.577170 9682 raft_consensus.cc:2229] T aad4288d6b1c49588682fd266b5e50fa P e3f506d848e746fa8c27d0e3c46b8379 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:17.577241 9683 raft_consensus.cc:2229] T aad4288d6b1c49588682fd266b5e50fa P 830f1c5539634e0b9645855927156696 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:17.577299 9682 raft_consensus.cc:2258] T aad4288d6b1c49588682fd266b5e50fa P e3f506d848e746fa8c27d0e3c46b8379 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:17.577351 9683 raft_consensus.cc:2258] T aad4288d6b1c49588682fd266b5e50fa P 830f1c5539634e0b9645855927156696 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:17.577472 9684 tablet_replica.cc:330] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411: stopping tablet replica
I20260928 06:45:17.577585 9684 raft_consensus.cc:2229] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:45:17.577875 9684 raft_consensus.cc:2258] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:17.578653 9684 ts_tablet_manager.cc:1905] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:17.578833 9682 ts_tablet_manager.cc:1905] T aad4288d6b1c49588682fd266b5e50fa P e3f506d848e746fa8c27d0e3c46b8379: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:17.578926 9092 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:45:17.579312 9092 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:45:17.581240 9684 ts_tablet_manager.cc:1918] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:45:17.581341 9684 log.cc:1196] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/aad4288d6b1c49588682fd266b5e50fa
I20260928 06:45:17.581336 9682 ts_tablet_manager.cc:1918] T aad4288d6b1c49588682fd266b5e50fa P e3f506d848e746fa8c27d0e3c46b8379: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:45:17.581403 9682 log.cc:1196] T aad4288d6b1c49588682fd266b5e50fa P e3f506d848e746fa8c27d0e3c46b8379: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-0/wal/wals/aad4288d6b1c49588682fd266b5e50fa
I20260928 06:45:17.581629 9684 ts_tablet_manager.cc:1939] T aad4288d6b1c49588682fd266b5e50fa P 387ce6f9361a439f95ede8adcce0c411: Deleting consensus metadata
I20260928 06:45:17.581673 9682 ts_tablet_manager.cc:1939] T aad4288d6b1c49588682fd266b5e50fa P e3f506d848e746fa8c27d0e3c46b8379: Deleting consensus metadata
I20260928 06:45:17.582329 9080 catalog_manager.cc:4841] TS 387ce6f9361a439f95ede8adcce0c411 (127.7.48.3:34929): tablet aad4288d6b1c49588682fd266b5e50fa (table test-table [id=dc3b722800b740f2b1a03ed6f56877de]) successfully deleted
I20260928 06:45:17.582636 9080 catalog_manager.cc:4841] TS e3f506d848e746fa8c27d0e3c46b8379 (127.7.48.1:46737): tablet aad4288d6b1c49588682fd266b5e50fa (table test-table [id=dc3b722800b740f2b1a03ed6f56877de]) successfully deleted
I20260928 06:45:17.585436 9231 tablet_service.cc:1467] Processing CreateTablet for tablet ed51da8e9590472d89ec6a9ac9e01364 (DEFAULT_TABLE table=test-table [id=3f1b7ad7fd654633b84974465423b174]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:17.585700 9231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed51da8e9590472d89ec6a9ac9e01364. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:17.585646 9371 tablet_service.cc:1467] Processing CreateTablet for tablet ed51da8e9590472d89ec6a9ac9e01364 (DEFAULT_TABLE table=test-table [id=3f1b7ad7fd654633b84974465423b174]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:17.585922 9371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed51da8e9590472d89ec6a9ac9e01364. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:17.587926 9693 tablet_bootstrap.cc:492] T ed51da8e9590472d89ec6a9ac9e01364 P e3f506d848e746fa8c27d0e3c46b8379: Bootstrap starting.
I20260928 06:45:17.588130 9694 tablet_bootstrap.cc:492] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696: Bootstrap starting.
I20260928 06:45:17.588411 9683 ts_tablet_manager.cc:1905] T aad4288d6b1c49588682fd266b5e50fa P 830f1c5539634e0b9645855927156696: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:17.588919 9693 tablet_bootstrap.cc:654] T ed51da8e9590472d89ec6a9ac9e01364 P e3f506d848e746fa8c27d0e3c46b8379: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:17.589058 9694 tablet_bootstrap.cc:654] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:17.590176 9693 tablet_bootstrap.cc:492] T ed51da8e9590472d89ec6a9ac9e01364 P e3f506d848e746fa8c27d0e3c46b8379: No bootstrap required, opened a new log
I20260928 06:45:17.590127 9510 tablet_service.cc:1467] Processing CreateTablet for tablet ed51da8e9590472d89ec6a9ac9e01364 (DEFAULT_TABLE table=test-table [id=3f1b7ad7fd654633b84974465423b174]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:17.590257 9693 ts_tablet_manager.cc:1397] T ed51da8e9590472d89ec6a9ac9e01364 P e3f506d848e746fa8c27d0e3c46b8379: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260928 06:45:17.590400 9510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed51da8e9590472d89ec6a9ac9e01364. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:17.590526 9683 ts_tablet_manager.cc:1918] T aad4288d6b1c49588682fd266b5e50fa P 830f1c5539634e0b9645855927156696: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:45:17.590591 9683 log.cc:1196] T aad4288d6b1c49588682fd266b5e50fa P 830f1c5539634e0b9645855927156696: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/aad4288d6b1c49588682fd266b5e50fa
I20260928 06:45:17.590664 9693 raft_consensus.cc:348] T ed51da8e9590472d89ec6a9ac9e01364 P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:17.590790 9693 raft_consensus.cc:374] T ed51da8e9590472d89ec6a9ac9e01364 P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:17.590826 9693 raft_consensus.cc:729] T ed51da8e9590472d89ec6a9ac9e01364 P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f506d848e746fa8c27d0e3c46b8379, State: Initialized, Role: FOLLOWER
I20260928 06:45:17.590855 9683 ts_tablet_manager.cc:1939] T aad4288d6b1c49588682fd266b5e50fa P 830f1c5539634e0b9645855927156696: Deleting consensus metadata
I20260928 06:45:17.590969 9693 consensus_queue.cc:260] T ed51da8e9590472d89ec6a9ac9e01364 P e3f506d848e746fa8c27d0e3c46b8379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:17.591233 9693 ts_tablet_manager.cc:1428] T ed51da8e9590472d89ec6a9ac9e01364 P e3f506d848e746fa8c27d0e3c46b8379: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:45:17.592847 9694 tablet_bootstrap.cc:492] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696: No bootstrap required, opened a new log
I20260928 06:45:17.592935 9694 ts_tablet_manager.cc:1397] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 06:45:17.592955 9695 tablet_bootstrap.cc:492] T ed51da8e9590472d89ec6a9ac9e01364 P 387ce6f9361a439f95ede8adcce0c411: Bootstrap starting.
I20260928 06:45:17.593360 9694 raft_consensus.cc:348] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:17.593483 9694 raft_consensus.cc:374] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:17.593516 9694 raft_consensus.cc:729] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830f1c5539634e0b9645855927156696, State: Initialized, Role: FOLLOWER
I20260928 06:45:17.593652 9694 consensus_queue.cc:260] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:17.593783 9695 tablet_bootstrap.cc:654] T ed51da8e9590472d89ec6a9ac9e01364 P 387ce6f9361a439f95ede8adcce0c411: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:17.593909 9694 ts_tablet_manager.cc:1428] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:45:17.594460 9080 catalog_manager.cc:4841] TS 830f1c5539634e0b9645855927156696 (127.7.48.2:32827): tablet aad4288d6b1c49588682fd266b5e50fa (table test-table [id=dc3b722800b740f2b1a03ed6f56877de]) successfully deleted
I20260928 06:45:17.595659 9695 tablet_bootstrap.cc:492] T ed51da8e9590472d89ec6a9ac9e01364 P 387ce6f9361a439f95ede8adcce0c411: No bootstrap required, opened a new log
I20260928 06:45:17.595780 9695 ts_tablet_manager.cc:1397] T ed51da8e9590472d89ec6a9ac9e01364 P 387ce6f9361a439f95ede8adcce0c411: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 06:45:17.596262 9695 raft_consensus.cc:348] T ed51da8e9590472d89ec6a9ac9e01364 P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:17.596395 9695 raft_consensus.cc:374] T ed51da8e9590472d89ec6a9ac9e01364 P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:17.596431 9695 raft_consensus.cc:729] T ed51da8e9590472d89ec6a9ac9e01364 P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 387ce6f9361a439f95ede8adcce0c411, State: Initialized, Role: FOLLOWER
I20260928 06:45:17.596550 9695 consensus_queue.cc:260] T ed51da8e9590472d89ec6a9ac9e01364 P 387ce6f9361a439f95ede8adcce0c411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:17.596840 9695 ts_tablet_manager.cc:1428] T ed51da8e9590472d89ec6a9ac9e01364 P 387ce6f9361a439f95ede8adcce0c411: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:45:17.890172 9699 raft_consensus.cc:482] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:45:17.890313 9699 raft_consensus.cc:504] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:17.891374 9699 leader_election.cc:288] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3f506d848e746fa8c27d0e3c46b8379 (127.7.48.1:46737), 387ce6f9361a439f95ede8adcce0c411 (127.7.48.3:34929)
I20260928 06:45:17.896749 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:17.892248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:49719 (local address 127.7.48.3:34929)
0928 06:45:17.892408 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:17.892413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:17.892427 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:45:17.892571 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:17.892573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:17.892609 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:17.892667 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:17.892955 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:17.893679 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:17.895324 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:17.895990 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:17.896100 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:17.896164 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:17.896528 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 06:45:17.896622 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 06:45:17.896781 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:17.891823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:52729 (local address 127.7.48.1:46737)
0928 06:45:17.892151 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:17.892170 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:17.892187 (+ 17us) server_negotiation.cc:407] Connection header received
0928 06:45:17.892251 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:17.892254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:17.892313 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:17.892411 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:17.892739 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:17.893570 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:17.895324 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:17.896004 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:17.896100 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:17.896166 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:17.896528 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 06:45:17.896646 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":143,"threads_started":1}
I20260928 06:45:17.896773 9712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:17.891679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:46737 (local address 127.7.48.2:52729)
0928 06:45:17.891988 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:17.892001 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:17.892128 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:17.892439 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:17.892442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:17.892455 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:17.892606 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:17.892611 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:17.893678 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:17.893681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:17.895197 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:17.895203 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:17.896111 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:17.896115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:17.896313 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:17.896403 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 06:45:17.896482 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:45:17.896586 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":233,"thread_start_us":90,"threads_started":1}
I20260928 06:45:17.896776 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:17.892101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:34929 (local address 127.7.48.2:49719)
0928 06:45:17.892344 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:17.892354 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:17.892449 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:17.892708 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:17.892714 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:17.892734 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:17.892846 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:17.892851 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:17.893797 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:17.893800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:17.895171 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:17.895199 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:17.896111 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:17.896115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:17.896311 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:17.896405 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 06:45:17.896482 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 06:45:17.896586 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":132,"thread_start_us":104,"threads_started":1}
I20260928 06:45:17.897320 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed51da8e9590472d89ec6a9ac9e01364" candidate_uuid: "830f1c5539634e0b9645855927156696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3f506d848e746fa8c27d0e3c46b8379" is_pre_election: true
I20260928 06:45:17.897473 9251 raft_consensus.cc:2454] T ed51da8e9590472d89ec6a9ac9e01364 P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830f1c5539634e0b9645855927156696 in term 0.
I20260928 06:45:17.897477 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed51da8e9590472d89ec6a9ac9e01364" candidate_uuid: "830f1c5539634e0b9645855927156696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "387ce6f9361a439f95ede8adcce0c411" is_pre_election: true
I20260928 06:45:17.897626 9531 raft_consensus.cc:2454] T ed51da8e9590472d89ec6a9ac9e01364 P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830f1c5539634e0b9645855927156696 in term 0.
I20260928 06:45:17.897853 9324 leader_election.cc:302] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830f1c5539634e0b9645855927156696, e3f506d848e746fa8c27d0e3c46b8379; no voters:
I20260928 06:45:17.898185 9699 raft_consensus.cc:2781] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:45:17.898242 9699 raft_consensus.cc:482] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:45:17.898269 9699 raft_consensus.cc:3037] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:17.899358 9699 raft_consensus.cc:504] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:17.899695 9699 leader_election.cc:288] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696 [CANDIDATE]: Term 1 election: Requested vote from peers e3f506d848e746fa8c27d0e3c46b8379 (127.7.48.1:46737), 387ce6f9361a439f95ede8adcce0c411 (127.7.48.3:34929)
I20260928 06:45:17.900101 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed51da8e9590472d89ec6a9ac9e01364" candidate_uuid: "830f1c5539634e0b9645855927156696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3f506d848e746fa8c27d0e3c46b8379"
I20260928 06:45:17.900157 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed51da8e9590472d89ec6a9ac9e01364" candidate_uuid: "830f1c5539634e0b9645855927156696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "387ce6f9361a439f95ede8adcce0c411"
I20260928 06:45:17.900230 9251 raft_consensus.cc:3037] T ed51da8e9590472d89ec6a9ac9e01364 P e3f506d848e746fa8c27d0e3c46b8379 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:17.900279 9531 raft_consensus.cc:3037] T ed51da8e9590472d89ec6a9ac9e01364 P 387ce6f9361a439f95ede8adcce0c411 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:17.901365 9531 raft_consensus.cc:2454] T ed51da8e9590472d89ec6a9ac9e01364 P 387ce6f9361a439f95ede8adcce0c411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830f1c5539634e0b9645855927156696 in term 1.
I20260928 06:45:17.901365 9251 raft_consensus.cc:2454] T ed51da8e9590472d89ec6a9ac9e01364 P e3f506d848e746fa8c27d0e3c46b8379 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830f1c5539634e0b9645855927156696 in term 1.
I20260928 06:45:17.901690 9324 leader_election.cc:302] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 387ce6f9361a439f95ede8adcce0c411, 830f1c5539634e0b9645855927156696; no voters:
I20260928 06:45:17.901878 9699 raft_consensus.cc:2781] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:17.902125 9699 raft_consensus.cc:686] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696 [term 1 LEADER]: Becoming Leader. State: Replica: 830f1c5539634e0b9645855927156696, State: Running, Role: LEADER
I20260928 06:45:17.902401 9699 consensus_queue.cc:237] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } }
I20260928 06:45:17.904520 9094 catalog_manager.cc:5495] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696 reported cstate change: term changed from 0 to 1, leader changed from <none> to 830f1c5539634e0b9645855927156696 (127.7.48.2). New cstate: current_term: 1 leader_uuid: "830f1c5539634e0b9645855927156696" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "830f1c5539634e0b9645855927156696" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 32827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 } health_report { overall_health: UNKNOWN } } }
I20260928 06:45:17.969846 9717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:17.966498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43958 (local address 127.7.48.2:32827)
0928 06:45:17.966791 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:17.966798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:17.966809 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:45:17.966875 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:17.966877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:17.966925 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:17.967024 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:45:17.967212 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:17.967771 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:17.968537 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:17.968769 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:17.968846 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:17.969058 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:45:17.969504 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:45:17.969608 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:17.969668 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 06:45:17.969715 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":151,"threads_started":1}
I20260928 06:45:17.979327 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:17.976616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:45253 (local address 127.7.48.3:34929)
0928 06:45:17.976757 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:17.976762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:17.976781 (+ 19us) server_negotiation.cc:407] Connection header received
0928 06:45:17.976897 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:17.976900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:17.976963 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:17.977060 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:45:17.977233 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:17.977704 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:17.978479 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:17.978690 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:17.978769 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:17.979006 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:45:17.979063 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:45:17.979114 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:17.979187 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 06:45:17.979228 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 06:45:17.979360 9720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:17.976424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:34929 (local address 127.7.48.2:45253)
0928 06:45:17.976732 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:17.976744 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:17.976816 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:17.977064 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:17.977066 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:17.977072 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:45:17.977159 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:17.977163 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:17.977790 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:17.977792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:17.978377 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:17.978384 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:17.978799 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:17.978802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:17.978864 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:17.978902 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:45:17.979129 (+ 227us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:45:17.979133 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:45:17.979180 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 06:45:17.979239 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":98,"threads_started":1}
I20260928 06:45:17.985154 9531 raft_consensus.cc:1261] T ed51da8e9590472d89ec6a9ac9e01364 P 387ce6f9361a439f95ede8adcce0c411 [term 1 FOLLOWER]: Refusing update from remote peer 830f1c5539634e0b9645855927156696: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:17.985160 9251 raft_consensus.cc:1261] T ed51da8e9590472d89ec6a9ac9e01364 P e3f506d848e746fa8c27d0e3c46b8379 [term 1 FOLLOWER]: Refusing update from remote peer 830f1c5539634e0b9645855927156696: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:17.985595 9715 consensus_queue.cc:1035] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f506d848e746fa8c27d0e3c46b8379" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 46737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:17.985764 9699 consensus_queue.cc:1035] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696 [LEADER]: Connected to new peer: Peer: permanent_uuid: "387ce6f9361a439f95ede8adcce0c411" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:18.001748 9717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:17.998463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:45355 (local address 127.7.48.2:32827)
0928 06:45:17.998785 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:17.998788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:17.998797 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:45:17.998829 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:17.998831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:17.998864 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:17.998944 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:45:17.999214 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:17.999726 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:18.000715 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:18.000953 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:18.001073 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:18.001329 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:45:18.001409 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:45:18.001480 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:18.001567 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 06:45:18.001615 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263}
I20260928 06:45:18.001749 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:17.998422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:32827 (local address 127.7.48.3:45355)
0928 06:45:17.998587 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:17.998601 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:17.998673 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:17.998955 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:17.998959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:17.998966 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:45:17.999060 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:17.999084 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:17.999825 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:17.999828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:18.000594 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:18.000611 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:18.001065 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:18.001068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:18.001168 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:18.001200 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:45:18.001486 (+ 286us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:45:18.001494 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 06:45:18.001557 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 06:45:18.001640 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
W20260928 06:45:18.002478 9664 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 06:45:19.557813 9094 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:19.557961 9094 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:19.560297 9094 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f4ac496c93b4f55b7c160bfa30fcad7: Sending DeleteTablet for 3 replicas of tablet ed51da8e9590472d89ec6a9ac9e01364
I20260928 06:45:19.560882 9371 tablet_service.cc:1514] Processing DeleteTablet for tablet ed51da8e9590472d89ec6a9ac9e01364 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:19 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:43950
I20260928 06:45:19.560928 9231 tablet_service.cc:1514] Processing DeleteTablet for tablet ed51da8e9590472d89ec6a9ac9e01364 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:19 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:59014
I20260928 06:45:19.560932 9510 tablet_service.cc:1514] Processing DeleteTablet for tablet ed51da8e9590472d89ec6a9ac9e01364 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:19 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:36558
I20260928 06:45:19.561250 9748 tablet_replica.cc:330] T ed51da8e9590472d89ec6a9ac9e01364 P 387ce6f9361a439f95ede8adcce0c411: stopping tablet replica
I20260928 06:45:19.561252 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 9166
I20260928 06:45:19.561897 9746 tablet_replica.cc:330] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696: stopping tablet replica
I20260928 06:45:19.561969 9748 raft_consensus.cc:2229] T ed51da8e9590472d89ec6a9ac9e01364 P 387ce6f9361a439f95ede8adcce0c411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:19.562024 9746 raft_consensus.cc:2229] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:45:19.562084 9748 raft_consensus.cc:2258] T ed51da8e9590472d89ec6a9ac9e01364 P 387ce6f9361a439f95ede8adcce0c411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:19.562331 9746 raft_consensus.cc:2258] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:19.562736 9748 ts_tablet_manager.cc:1905] T ed51da8e9590472d89ec6a9ac9e01364 P 387ce6f9361a439f95ede8adcce0c411: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:19.562883 9746 ts_tablet_manager.cc:1905] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:19.565495 9748 ts_tablet_manager.cc:1918] T ed51da8e9590472d89ec6a9ac9e01364 P 387ce6f9361a439f95ede8adcce0c411: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:45:19.565495 9746 ts_tablet_manager.cc:1918] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:45:19.565575 9746 log.cc:1196] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/ed51da8e9590472d89ec6a9ac9e01364
I20260928 06:45:19.565575 9748 log.cc:1196] T ed51da8e9590472d89ec6a9ac9e01364 P 387ce6f9361a439f95ede8adcce0c411: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/ed51da8e9590472d89ec6a9ac9e01364
I20260928 06:45:19.565891 9746 ts_tablet_manager.cc:1939] T ed51da8e9590472d89ec6a9ac9e01364 P 830f1c5539634e0b9645855927156696: Deleting consensus metadata
I20260928 06:45:19.565891 9748 ts_tablet_manager.cc:1939] T ed51da8e9590472d89ec6a9ac9e01364 P 387ce6f9361a439f95ede8adcce0c411: Deleting consensus metadata
I20260928 06:45:19.566517 9080 catalog_manager.cc:4841] TS 387ce6f9361a439f95ede8adcce0c411 (127.7.48.3:34929): tablet ed51da8e9590472d89ec6a9ac9e01364 (table test-table [id=3f1b7ad7fd654633b84974465423b174]) successfully deleted
I20260928 06:45:19.566677 9080 catalog_manager.cc:4841] TS 830f1c5539634e0b9645855927156696 (127.7.48.2:32827): tablet ed51da8e9590472d89ec6a9ac9e01364 (table test-table [id=3f1b7ad7fd654633b84974465423b174]) successfully deleted
W20260928 06:45:19.572082 9080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.48.1:46737 (error 108)
I20260928 06:45:19.572368 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 9306
I20260928 06:45:19.574319 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:19.573862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:46737 (local address 127.0.0.1:59038)
0928 06:45:19.574099 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:19.574193 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.1:46737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
W20260928 06:45:19.574698 9080 catalog_manager.cc:4584] TS e3f506d848e746fa8c27d0e3c46b8379 (127.7.48.1:46737): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ed51da8e9590472d89ec6a9ac9e01364: Network error: Client connection negotiation failed: client connection to 127.7.48.1:46737: connect: Connection refused (error 111)
I20260928 06:45:19.579285 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 9446
I20260928 06:45:19.585830 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 9062
2026-09-28T06:45:19Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6643 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:45:19 dist-test-slave-twl6 krb5kdc[9753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:45:19 dist-test-slave-twl6 krb5kdc[9753](info): set up 2 sockets
Sep 28 06:45:19 dist-test-slave-twl6 krb5kdc[9753](info): commencing operation
krb5kdc: starting...
W20260928 06:45:20.632400 7360 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:45:20 dist-test-slave-twl6 krb5kdc[9753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:45:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:45:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:45:20.750489 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:36081
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:33117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:36081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:20.857452 9766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:20.857707 9766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:20.857774 9766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:20.861752 9766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:45:20.861826 9766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:20.861862 9766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:20.861914 9766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:45:20.861950 9766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:45:20.866981 9766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:33117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:36081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:36081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:20.867998 9766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:20.868952 9766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:45:20.876963 9766 server_base.cc:1034] running on GCE node
W20260928 06:45:20.876977 9771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:20.877105 9774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:20.877022 9772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:20.877710 9766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:20.878667 9766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:20.879842 9766 hybrid_clock.cc:648] HybridClock initialized: now 1790577920879833 us; error 41 us; skew 500 ppm
Sep 28 06:45:20 dist-test-slave-twl6 krb5kdc[9753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577920, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:20.882488 9766 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:45:20.883340 9766 webserver.cc:460] Webserver started at http://127.7.48.62:36763/ using document root <none> and password file <none>
I20260928 06:45:20.883925 9766 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:20.883997 9766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:20.884207 9766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:20.886093 9766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "64737782aa79495188889df59ca5de4e"
format_stamp: "Formatted at 2026-09-28 06:45:20 on dist-test-slave-twl6"
server_key: "377bd4213879dc6e86730f3823f41481"
server_key_iv: "86481da887d777393711d6e94bb1c3b7"
server_key_version: "encryptionkey@0"
I20260928 06:45:20.886639 9766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "64737782aa79495188889df59ca5de4e"
format_stamp: "Formatted at 2026-09-28 06:45:20 on dist-test-slave-twl6"
server_key: "377bd4213879dc6e86730f3823f41481"
server_key_iv: "86481da887d777393711d6e94bb1c3b7"
server_key_version: "encryptionkey@0"
I20260928 06:45:20.890184 9766 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 06:45:20.892527 9781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:20.893680 9766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:45:20.893798 9766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "64737782aa79495188889df59ca5de4e"
format_stamp: "Formatted at 2026-09-28 06:45:20 on dist-test-slave-twl6"
server_key: "377bd4213879dc6e86730f3823f41481"
server_key_iv: "86481da887d777393711d6e94bb1c3b7"
server_key_version: "encryptionkey@0"
I20260928 06:45:20.893893 9766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:20.927074 9766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:20.930241 9766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:20.930438 9766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:20.937532 9766 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:36081
I20260928 06:45:20.937556 9833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:36081 every 8 connection(s)
I20260928 06:45:20.938499 9766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:45:20.941308 9834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:20.945968 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 9766
I20260928 06:45:20.946094 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:45:20.946393 7360 external_mini_cluster.cc:1428] Setting key 1d51fe0b1253f644ac59251209de3eab
I20260928 06:45:20.946959 9834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64737782aa79495188889df59ca5de4e: Bootstrap starting.
I20260928 06:45:20.949463 9834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64737782aa79495188889df59ca5de4e: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:20.950196 9834 log.cc:824] T 00000000000000000000000000000000 P 64737782aa79495188889df59ca5de4e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:20.952005 9834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64737782aa79495188889df59ca5de4e: No bootstrap required, opened a new log
Sep 28 06:45:20 dist-test-slave-twl6 krb5kdc[9753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:20.956120 9834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64737782aa79495188889df59ca5de4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64737782aa79495188889df59ca5de4e" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36081 } }
I20260928 06:45:20.956382 9834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64737782aa79495188889df59ca5de4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:20.956467 9834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64737782aa79495188889df59ca5de4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64737782aa79495188889df59ca5de4e, State: Initialized, Role: FOLLOWER
I20260928 06:45:20.956951 9834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64737782aa79495188889df59ca5de4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64737782aa79495188889df59ca5de4e" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36081 } }
I20260928 06:45:20.957079 9834 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64737782aa79495188889df59ca5de4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:45:20.957178 9834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64737782aa79495188889df59ca5de4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:45:20.957288 9834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64737782aa79495188889df59ca5de4e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:20.958251 9834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64737782aa79495188889df59ca5de4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64737782aa79495188889df59ca5de4e" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36081 } }
I20260928 06:45:20.958621 9834 leader_election.cc:302] T 00000000000000000000000000000000 P 64737782aa79495188889df59ca5de4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64737782aa79495188889df59ca5de4e; no voters:
I20260928 06:45:20.958923 9834 leader_election.cc:288] T 00000000000000000000000000000000 P 64737782aa79495188889df59ca5de4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:45:20.959007 9839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64737782aa79495188889df59ca5de4e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:20.959250 9839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64737782aa79495188889df59ca5de4e [term 1 LEADER]: Becoming Leader. State: Replica: 64737782aa79495188889df59ca5de4e, State: Running, Role: LEADER
I20260928 06:45:20.959568 9839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64737782aa79495188889df59ca5de4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64737782aa79495188889df59ca5de4e" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36081 } }
I20260928 06:45:20.960137 9834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64737782aa79495188889df59ca5de4e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:45:20.960763 9839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64737782aa79495188889df59ca5de4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64737782aa79495188889df59ca5de4e. Latest consensus state: current_term: 1 leader_uuid: "64737782aa79495188889df59ca5de4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64737782aa79495188889df59ca5de4e" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36081 } } }
I20260928 06:45:20.961202 9837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:20.947802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40116 (local address 127.7.48.62:36081)
0928 06:45:20.948307 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:20.948322 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:20.948357 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:45:20.948953 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:20.948969 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:20.949296 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:20.949643 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:20.950479 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:20.951414 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:20.952385 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:20.952841 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:20.953002 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:20.955725 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:20.955730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:20.955742 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:20.955767 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:20.957667 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:20.958138 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:20.958143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:20.958146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:20.958246 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:20.958521 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:20.958523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:20.958525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:20.958968 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:20.959161 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:20.959868 (+ 707us) server_negotiation.cc:299] Negotiation successful
0928 06:45:20.960175 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":170,"threads_started":1}
I20260928 06:45:20.961520 9839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64737782aa79495188889df59ca5de4e [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:20.961400 9840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64737782aa79495188889df59ca5de4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64737782aa79495188889df59ca5de4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64737782aa79495188889df59ca5de4e" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36081 } } }
I20260928 06:45:20.961711 9840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64737782aa79495188889df59ca5de4e [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:20.962121 9848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:45:20.965173 9848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:45:20.970796 9848 catalog_manager.cc:1347] Generated new cluster ID: 7c8ea35e64bf4ee8be17711125dbb78a
I20260928 06:45:20.970861 9848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:45:20.989352 9848 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:45:20.990398 9848 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:45:20.996085 9848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64737782aa79495188889df59ca5de4e: Generated new TSK 0
I20260928 06:45:20.996666 9848 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
I20260928 06:45:21.026805 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:36081
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:33117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:21.170918 9860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:21.171185 9860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:21.171226 9860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:21.175091 9860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:21.175145 9860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:21.175203 9860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:45:21.180186 9860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:33117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:36081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:21.181083 9860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:21.182042 9860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:21.191279 9866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:21.191308 9868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:21.191290 9865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:21.191717 9860 server_base.cc:1034] running on GCE node
I20260928 06:45:21.192167 9860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:21.192793 9860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:21.193970 9860 hybrid_clock.cc:648] HybridClock initialized: now 1790577921193944 us; error 45 us; skew 500 ppm
Sep 28 06:45:21 dist-test-slave-twl6 krb5kdc[9753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577921, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:21.196789 9860 init.cc:377] Logged in from keytab as kudu/127.7.48.1@KRBTEST.COM (short username kudu)
I20260928 06:45:21.197629 9860 webserver.cc:460] Webserver started at http://127.7.48.1:37793/ using document root <none> and password file <none>
I20260928 06:45:21.198230 9860 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:21.198321 9860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:21.198515 9860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:21.200961 9860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "af3afbaccf224cf78295529fccf416c0"
format_stamp: "Formatted at 2026-09-28 06:45:21 on dist-test-slave-twl6"
server_key: "1e2669442533ed6e24fb3537d3b7762b"
server_key_iv: "70a2a5268a7f250650af474750c1841a"
server_key_version: "encryptionkey@0"
I20260928 06:45:21.201731 9860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "af3afbaccf224cf78295529fccf416c0"
format_stamp: "Formatted at 2026-09-28 06:45:21 on dist-test-slave-twl6"
server_key: "1e2669442533ed6e24fb3537d3b7762b"
server_key_iv: "70a2a5268a7f250650af474750c1841a"
server_key_version: "encryptionkey@0"
I20260928 06:45:21.206149 9860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:45:21.208657 9875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:21.209853 9860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:21.209961 9860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "af3afbaccf224cf78295529fccf416c0"
format_stamp: "Formatted at 2026-09-28 06:45:21 on dist-test-slave-twl6"
server_key: "1e2669442533ed6e24fb3537d3b7762b"
server_key_iv: "70a2a5268a7f250650af474750c1841a"
server_key_version: "encryptionkey@0"
I20260928 06:45:21.210048 9860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:21.229930 9860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:21.233352 9860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:21.233520 9860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:21.234063 9860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:21.234845 9860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:21.234892 9860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:21.234931 9860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:21.234962 9860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:21.243628 9860 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:35961
I20260928 06:45:21.243666 9988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:35961 every 8 connection(s)
I20260928 06:45:21.244552 9860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:45:21.253620 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 9860
I20260928 06:45:21.253712 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
Sep 28 06:45:21 dist-test-slave-twl6 krb5kdc[9753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577921, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:21.253983 7360 external_mini_cluster.cc:1428] Setting key 340c436e0f19c7440ed11f1df99d5c01
I20260928 06:45:21.258389 9837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:21.246401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:43665 (local address 127.7.48.62:36081)
0928 06:45:21.246531 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:21.246535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:21.247404 (+ 869us) server_negotiation.cc:407] Connection header received
0928 06:45:21.248973 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:21.248977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:21.249028 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:21.249107 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:21.250191 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:21.250738 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:21.251792 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:21.252080 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:21.252161 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:21.254547 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:21.254551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:21.254554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:21.254586 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:21.256298 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:21.256772 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:21.256779 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:21.256783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:21.256882 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:21.257297 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:21.257299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:21.257300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:21.257477 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:21.257562 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:21.258164 (+ 602us) server_negotiation.cc:299] Negotiation successful
0928 06:45:21.258270 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 06:45:21.259302 9991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:21.246718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:36081 (local address 127.7.48.1:43665)
0928 06:45:21.247193 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:21.247262 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:21.248776 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:21.249286 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:21.249292 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:21.249617 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:21.250067 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:21.250079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:21.250840 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:21.250842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:21.251690 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:21.251698 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:21.252210 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:21.252213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:21.252349 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:21.252869 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:21.252885 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:21.254367 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:21.256410 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:21.256415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:21.256426 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:21.256676 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:21.256974 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:21.256976 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:21.256978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:21.257184 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:21.257583 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:21.257591 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:21.257903 (+ 312us) client_negotiation.cc:769] Sending connection context
0928 06:45:21.258122 (+ 219us) client_negotiation.cc:240] Negotiation successful
0928 06:45:21.258496 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":147,"threads_started":1}
I20260928 06:45:21.260587 9989 heartbeater.cc:347] Connected to a master server at 127.7.48.62:36081
I20260928 06:45:21.260824 9989 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:21.261340 9989 heartbeater.cc:511] Master 127.7.48.62:36081 requested a full tablet report, sending...
I20260928 06:45:21.262847 9798 ts_manager.cc:194] Registered new tserver with Master: af3afbaccf224cf78295529fccf416c0 (127.7.48.1:35961)
WARNING: no policy specified for kudu/127.7.48.2@KRBTEST.COM; defaulting to no policy
I20260928 06:45:21.264043 9798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.1@KRBTEST.COM'} at 127.7.48.1:43665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
I20260928 06:45:21.280988 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:36081
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:33117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:21.391398 9994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:21.391662 9994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:21.391705 9994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:21.395709 9994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:21.395766 9994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:21.395821 9994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:45:21.401333 9994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:33117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:36081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:21.402470 9994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:21.403452 9994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:21.413167 9999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:21.413223 10002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:21.413167 10000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:21.413601 9994 server_base.cc:1034] running on GCE node
I20260928 06:45:21.414001 9994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:21.414577 9994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:21.415732 9994 hybrid_clock.cc:648] HybridClock initialized: now 1790577921415717 us; error 29 us; skew 500 ppm
Sep 28 06:45:21 dist-test-slave-twl6 krb5kdc[9753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577921, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:21.418437 9994 init.cc:377] Logged in from keytab as kudu/127.7.48.2@KRBTEST.COM (short username kudu)
I20260928 06:45:21.419247 9994 webserver.cc:460] Webserver started at http://127.7.48.2:41425/ using document root <none> and password file <none>
I20260928 06:45:21.419759 9994 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:21.419813 9994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:21.420047 9994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:21.422055 9994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "aa04d88580284c74bee783de95de008b"
format_stamp: "Formatted at 2026-09-28 06:45:21 on dist-test-slave-twl6"
server_key: "14a019b38d2dbfde20da0695ec47bd7c"
server_key_iv: "8c91907406b2df7225998ec8fa422802"
server_key_version: "encryptionkey@0"
I20260928 06:45:21.422561 9994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "aa04d88580284c74bee783de95de008b"
format_stamp: "Formatted at 2026-09-28 06:45:21 on dist-test-slave-twl6"
server_key: "14a019b38d2dbfde20da0695ec47bd7c"
server_key_iv: "8c91907406b2df7225998ec8fa422802"
server_key_version: "encryptionkey@0"
I20260928 06:45:21.426246 9994 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 06:45:21.428705 10009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:21.430065 9994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260928 06:45:21.430188 9994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "aa04d88580284c74bee783de95de008b"
format_stamp: "Formatted at 2026-09-28 06:45:21 on dist-test-slave-twl6"
server_key: "14a019b38d2dbfde20da0695ec47bd7c"
server_key_iv: "8c91907406b2df7225998ec8fa422802"
server_key_version: "encryptionkey@0"
I20260928 06:45:21.430289 9994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:21.443753 9994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:21.446529 9994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:21.446753 9994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:21.447376 9994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:21.448364 9994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:21.448422 9994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:21.448486 9994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:21.448508 9994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:21.457050 9994 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:42847
I20260928 06:45:21.457070 10122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:42847 every 8 connection(s)
I20260928 06:45:21.457942 9994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:45:21 dist-test-slave-twl6 krb5kdc[9753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577921, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:21.467228 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 9994
I20260928 06:45:21.467346 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:45:21.467636 7360 external_mini_cluster.cc:1428] Setting key 3e8a3399a70795f40af02cbfc66d9756
I20260928 06:45:21.471403 9837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:21.459636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:42147 (local address 127.7.48.62:36081)
0928 06:45:21.459798 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:21.459805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:21.460869 (+ 1064us) server_negotiation.cc:407] Connection header received
0928 06:45:21.461844 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:21.461847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:21.461892 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:21.461965 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:21.463106 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:21.463684 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:21.464843 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:21.465069 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:21.465160 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:21.467580 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:21.467584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:21.467587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:21.467615 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:21.469121 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:21.469742 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:21.469745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:21.469746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:21.469818 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:21.470196 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:21.470198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:21.470199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:21.470359 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:21.470427 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:21.471060 (+ 633us) server_negotiation.cc:299] Negotiation successful
0928 06:45:21.471198 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:45:21.472260 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:21.460071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:36081 (local address 127.7.48.2:42147)
0928 06:45:21.460744 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:21.460785 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:21.461630 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:21.462144 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:21.462151 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:21.462506 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:21.462973 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:21.462985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:21.463823 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:21.463826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:21.464733 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:21.464741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:21.465224 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:21.465226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:21.465361 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:21.465958 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:21.465974 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:21.467394 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:21.469229 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:21.469235 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:21.469268 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:21.469648 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:21.469900 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:21.469905 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:21.469910 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:21.470104 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:21.470449 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:21.470459 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:21.470801 (+ 342us) client_negotiation.cc:769] Sending connection context
0928 06:45:21.470970 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 06:45:21.471211 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":111,"threads_started":1}
I20260928 06:45:21.473448 10123 heartbeater.cc:347] Connected to a master server at 127.7.48.62:36081
I20260928 06:45:21.473723 10123 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:21.474334 10123 heartbeater.cc:511] Master 127.7.48.62:36081 requested a full tablet report, sending...
I20260928 06:45:21.475584 9798 ts_manager.cc:194] Registered new tserver with Master: aa04d88580284c74bee783de95de008b (127.7.48.2:42847)
I20260928 06:45:21.476424 9798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.2@KRBTEST.COM'} at 127.7.48.2:42147
WARNING: no policy specified for kudu/127.7.48.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
I20260928 06:45:21.494962 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:36081
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:33117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:21.605403 10128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:21.605650 10128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:21.605692 10128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:21.609683 10128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:21.609738 10128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:21.609797 10128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:45:21.615170 10128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:33117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:36081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:21.616187 10128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:21.617130 10128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:21.626509 10136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:21.626504 10133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:21.626502 10134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:21.626760 10128 server_base.cc:1034] running on GCE node
I20260928 06:45:21.627153 10128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:21.627805 10128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:21.628985 10128 hybrid_clock.cc:648] HybridClock initialized: now 1790577921628968 us; error 28 us; skew 500 ppm
Sep 28 06:45:21 dist-test-slave-twl6 krb5kdc[9753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577921, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:21.631691 10128 init.cc:377] Logged in from keytab as kudu/127.7.48.3@KRBTEST.COM (short username kudu)
I20260928 06:45:21.632619 10128 webserver.cc:460] Webserver started at http://127.7.48.3:46537/ using document root <none> and password file <none>
I20260928 06:45:21.633229 10128 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:21.633316 10128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:21.633550 10128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:21.635639 10128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "7bc2aade752a4dd7a57f43cfe82a7f33"
format_stamp: "Formatted at 2026-09-28 06:45:21 on dist-test-slave-twl6"
server_key: "c9dd253a624126bac6328dc453437d39"
server_key_iv: "2f38256e68d1ebabf0469472c5a14b20"
server_key_version: "encryptionkey@0"
I20260928 06:45:21.636228 10128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "7bc2aade752a4dd7a57f43cfe82a7f33"
format_stamp: "Formatted at 2026-09-28 06:45:21 on dist-test-slave-twl6"
server_key: "c9dd253a624126bac6328dc453437d39"
server_key_iv: "2f38256e68d1ebabf0469472c5a14b20"
server_key_version: "encryptionkey@0"
I20260928 06:45:21.639972 10128 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:45:21.642359 10143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:21.643599 10128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:21.643736 10128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "7bc2aade752a4dd7a57f43cfe82a7f33"
format_stamp: "Formatted at 2026-09-28 06:45:21 on dist-test-slave-twl6"
server_key: "c9dd253a624126bac6328dc453437d39"
server_key_iv: "2f38256e68d1ebabf0469472c5a14b20"
server_key_version: "encryptionkey@0"
I20260928 06:45:21.643844 10128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:21.667081 10128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:21.669909 10128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:21.670104 10128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:21.670687 10128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:21.671535 10128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:21.671604 10128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:21.671685 10128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:21.671736 10128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:21.680560 10128 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:36403
I20260928 06:45:21.680586 10256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:36403 every 8 connection(s)
I20260928 06:45:21.681454 10128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:45:21 dist-test-slave-twl6 krb5kdc[9753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577921, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:21.691393 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 10128
I20260928 06:45:21.691524 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:45:21.691811 7360 external_mini_cluster.cc:1428] Setting key e3f70f10486b0c90ec18a7ee79695713
I20260928 06:45:21.694813 9837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:21.683144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:42043 (local address 127.7.48.62:36081)
0928 06:45:21.683278 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:21.683281 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:21.684465 (+ 1184us) server_negotiation.cc:407] Connection header received
0928 06:45:21.685400 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:21.685404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:21.685495 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:21.685598 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:21.686756 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:21.687275 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:21.688495 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:21.688722 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:21.688829 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:21.691170 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:21.691173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:21.691175 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:21.691200 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:21.692740 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:21.693330 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:21.693334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:21.693336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:21.693422 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:21.693782 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:21.693785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:21.693787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:21.693969 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:21.694081 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:21.694535 (+ 454us) server_negotiation.cc:299] Negotiation successful
0928 06:45:21.694661 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:45:21.695655 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:21.683533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:36081 (local address 127.7.48.3:42043)
0928 06:45:21.684319 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:21.684361 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:21.685174 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:21.685769 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:21.685776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:21.686118 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:21.686615 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:21.686628 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:21.687399 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:21.687407 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:21.688366 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:21.688375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:21.688881 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:21.688885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:21.689056 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:21.689623 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:21.689639 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:21.691005 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:21.692857 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:21.692861 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:21.692872 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:21.693223 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:21.693520 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:21.693524 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:21.693526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:21.693683 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:21.694074 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:21.694080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:21.694310 (+ 230us) client_negotiation.cc:769] Sending connection context
0928 06:45:21.694492 (+ 182us) client_negotiation.cc:240] Negotiation successful
0928 06:45:21.694815 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":118,"threads_started":1}
I20260928 06:45:21.696672 10257 heartbeater.cc:347] Connected to a master server at 127.7.48.62:36081
I20260928 06:45:21.696925 10257 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:21.697371 10257 heartbeater.cc:511] Master 127.7.48.62:36081 requested a full tablet report, sending...
I20260928 06:45:21.698313 9798 ts_manager.cc:194] Registered new tserver with Master: 7bc2aade752a4dd7a57f43cfe82a7f33 (127.7.48.3:36403)
I20260928 06:45:21.698804 9798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.3@KRBTEST.COM'} at 127.7.48.3:42043
I20260928 06:45:21.705407 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 06:45:21.717665 9837 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:21.716339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40132 (local address 127.7.48.62:36081)
0928 06:45:21.716481 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:21.716485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:21.716614 (+ 129us) server_negotiation.cc:407] Connection header received
0928 06:45:21.716720 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:21.716722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:21.716764 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:21.716833 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:21.717502 (+ 669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40132: BlockingRecv error: recv got EOF from 127.0.0.1:40132 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:45:21.718595 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 9860
I20260928 06:45:21.724217 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 9994
I20260928 06:45:21.729120 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 10128
I20260928 06:45:21.734441 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 9766
2026-09-28T06:45:21Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2147 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:45:21 dist-test-slave-twl6 krb5kdc[10269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:45:21 dist-test-slave-twl6 krb5kdc[10269](info): set up 2 sockets
Sep 28 06:45:21 dist-test-slave-twl6 krb5kdc[10269](info): commencing operation
krb5kdc: starting...
W20260928 06:45:22.768407 7360 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:45:22 dist-test-slave-twl6 krb5kdc[10269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:45:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:45:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:45:22.886116 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:43299
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:35053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:43299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:22.993829 10282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:22.994130 10282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:22.994202 10282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:22.998081 10282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:45:22.998158 10282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:22.998191 10282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:22.998241 10282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:45:22.998287 10282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:45:23.003283 10282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:35053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:43299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:43299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:23.004318 10282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:23.005286 10282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:23.013269 10288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:23.013302 10290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:23.013303 10287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:23.013535 10282 server_base.cc:1034] running on GCE node
I20260928 06:45:23.014170 10282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:23.015244 10282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:23.016400 10282 hybrid_clock.cc:648] HybridClock initialized: now 1790577923016379 us; error 32 us; skew 500 ppm
Sep 28 06:45:23 dist-test-slave-twl6 krb5kdc[10269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577923, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:23.024580 10282 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:45:23.025560 10282 webserver.cc:460] Webserver started at http://127.7.48.62:38187/ using document root <none> and password file <none>
I20260928 06:45:23.026135 10282 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:23.026196 10282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:23.026438 10282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:23.028568 10282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "15f82070ab4e48979c63cc8be8eb0ad9"
format_stamp: "Formatted at 2026-09-28 06:45:23 on dist-test-slave-twl6"
server_key: "7ec8ec6cf0f064edbe018bfb99c1475d"
server_key_iv: "5f6952bf3468eff77a53138dcb223dff"
server_key_version: "encryptionkey@0"
I20260928 06:45:23.029109 10282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "15f82070ab4e48979c63cc8be8eb0ad9"
format_stamp: "Formatted at 2026-09-28 06:45:23 on dist-test-slave-twl6"
server_key: "7ec8ec6cf0f064edbe018bfb99c1475d"
server_key_iv: "5f6952bf3468eff77a53138dcb223dff"
server_key_version: "encryptionkey@0"
I20260928 06:45:23.032794 10282 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:45:23.035317 10297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:23.036590 10282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:45:23.036755 10282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "15f82070ab4e48979c63cc8be8eb0ad9"
format_stamp: "Formatted at 2026-09-28 06:45:23 on dist-test-slave-twl6"
server_key: "7ec8ec6cf0f064edbe018bfb99c1475d"
server_key_iv: "5f6952bf3468eff77a53138dcb223dff"
server_key_version: "encryptionkey@0"
I20260928 06:45:23.036870 10282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:23.052553 10282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:23.055586 10282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:23.055819 10282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:23.062947 10282 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:43299
I20260928 06:45:23.063231 10349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:43299 every 8 connection(s)
I20260928 06:45:23.064011 10282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:45:23.066843 10350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:23.072306 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 10282
I20260928 06:45:23.072436 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:45:23.072507 10350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15f82070ab4e48979c63cc8be8eb0ad9: Bootstrap starting.
I20260928 06:45:23.072733 7360 external_mini_cluster.cc:1428] Setting key 54e2c646dada4ec7942ba1d1b3eb6d77
I20260928 06:45:23.075147 10350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15f82070ab4e48979c63cc8be8eb0ad9: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:23.075958 10350 log.cc:824] T 00000000000000000000000000000000 P 15f82070ab4e48979c63cc8be8eb0ad9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:23.077943 10350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15f82070ab4e48979c63cc8be8eb0ad9: No bootstrap required, opened a new log
Sep 28 06:45:23 dist-test-slave-twl6 krb5kdc[10269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:23.082281 10350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15f82070ab4e48979c63cc8be8eb0ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15f82070ab4e48979c63cc8be8eb0ad9" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 43299 } }
I20260928 06:45:23.082520 10350 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15f82070ab4e48979c63cc8be8eb0ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:23.082602 10350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15f82070ab4e48979c63cc8be8eb0ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15f82070ab4e48979c63cc8be8eb0ad9, State: Initialized, Role: FOLLOWER
I20260928 06:45:23.083065 10350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15f82070ab4e48979c63cc8be8eb0ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15f82070ab4e48979c63cc8be8eb0ad9" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 43299 } }
I20260928 06:45:23.083221 10350 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15f82070ab4e48979c63cc8be8eb0ad9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:45:23.083302 10350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15f82070ab4e48979c63cc8be8eb0ad9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:45:23.083401 10350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15f82070ab4e48979c63cc8be8eb0ad9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:23.084348 10350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15f82070ab4e48979c63cc8be8eb0ad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15f82070ab4e48979c63cc8be8eb0ad9" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 43299 } }
I20260928 06:45:23.084734 10350 leader_election.cc:302] T 00000000000000000000000000000000 P 15f82070ab4e48979c63cc8be8eb0ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15f82070ab4e48979c63cc8be8eb0ad9; no voters:
I20260928 06:45:23.085099 10350 leader_election.cc:288] T 00000000000000000000000000000000 P 15f82070ab4e48979c63cc8be8eb0ad9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:45:23.085234 10355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15f82070ab4e48979c63cc8be8eb0ad9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:23.085474 10355 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15f82070ab4e48979c63cc8be8eb0ad9 [term 1 LEADER]: Becoming Leader. State: Replica: 15f82070ab4e48979c63cc8be8eb0ad9, State: Running, Role: LEADER
I20260928 06:45:23.085840 10355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15f82070ab4e48979c63cc8be8eb0ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15f82070ab4e48979c63cc8be8eb0ad9" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 43299 } }
I20260928 06:45:23.086484 10350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15f82070ab4e48979c63cc8be8eb0ad9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:45:23.087613 10357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15f82070ab4e48979c63cc8be8eb0ad9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15f82070ab4e48979c63cc8be8eb0ad9. Latest consensus state: current_term: 1 leader_uuid: "15f82070ab4e48979c63cc8be8eb0ad9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15f82070ab4e48979c63cc8be8eb0ad9" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 43299 } } }
I20260928 06:45:23.087744 10357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15f82070ab4e48979c63cc8be8eb0ad9 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:23.087818 10353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:23.074013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33038 (local address 127.7.48.62:43299)
0928 06:45:23.074424 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:23.074443 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:23.074493 (+ 50us) server_negotiation.cc:407] Connection header received
0928 06:45:23.075153 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:23.075170 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:23.075569 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:23.075850 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:23.076783 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:23.077789 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:23.078913 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:23.079387 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:23.079514 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:23.082437 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:23.082442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:23.082455 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:23.082481 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:23.084449 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:23.084990 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:23.084999 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:23.085008 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:23.085157 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:23.085488 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:23.085491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:23.085493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:23.085986 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:23.086188 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:23.086473 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 06:45:23.086787 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":132,"threads_started":1}
I20260928 06:45:23.088090 10356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15f82070ab4e48979c63cc8be8eb0ad9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15f82070ab4e48979c63cc8be8eb0ad9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15f82070ab4e48979c63cc8be8eb0ad9" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 43299 } } }
I20260928 06:45:23.088169 10356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15f82070ab4e48979c63cc8be8eb0ad9 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:23.090768 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 06:45:23.091684 10370 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 15f82070ab4e48979c63cc8be8eb0ad9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:45:23.091765 10370 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:45:23.091843 10364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:45:23.092731 10364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:45:23.098865 10364 catalog_manager.cc:1347] Generated new cluster ID: c801afa9f49b4440af2d7b5188533fd6
I20260928 06:45:23.098963 10364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:45:23.105176 10364 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:45:23.106118 10364 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:45:23.108141 10353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:23.102849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33042 (local address 127.7.48.62:43299)
0928 06:45:23.102962 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:23.102966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:23.103186 (+ 220us) server_negotiation.cc:407] Connection header received
0928 06:45:23.103314 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:23.103317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:23.103368 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:23.103451 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:23.104063 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:23.104633 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:23.105769 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:23.106189 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:23.106284 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:23.107152 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:23.107156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:23.107159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:45:23.107652 (+ 493us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:23.107683 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:45:23.107701 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:23.107789 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:23.107922 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 06:45:23.107998 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":42}
I20260928 06:45:23.119221 10364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15f82070ab4e48979c63cc8be8eb0ad9: Generated new TSK 0
I20260928 06:45:23.120066 10364 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:45:23.125221 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 10282
2026-09-28T06:45:23Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1388 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:45:23 dist-test-slave-twl6 krb5kdc[10383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:45:23 dist-test-slave-twl6 krb5kdc[10383](info): set up 2 sockets
Sep 28 06:45:23 dist-test-slave-twl6 krb5kdc[10383](info): commencing operation
krb5kdc: starting...
W20260928 06:45:24.152400 7360 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:45:24 dist-test-slave-twl6 krb5kdc[10383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:45:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:45:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:45:24.275106 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:40171
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:40971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:40171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:24.388335 10396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:24.388629 10396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:24.388688 10396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:24.392712 10396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:45:24.392768 10396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:24.392788 10396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:24.392804 10396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:45:24.392846 10396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:45:24.398131 10396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:40971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:40171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:40171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:24.399236 10396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:24.400273 10396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:24.408432 10401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:24.408540 10396 server_base.cc:1034] running on GCE node
W20260928 06:45:24.408475 10402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:24.408512 10404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:24.409296 10396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:24.410748 10396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:24.411933 10396 hybrid_clock.cc:648] HybridClock initialized: now 1790577924411907 us; error 41 us; skew 500 ppm
Sep 28 06:45:24 dist-test-slave-twl6 krb5kdc[10383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577924, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:24.414599 10396 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:45:24.415434 10396 webserver.cc:460] Webserver started at http://127.7.48.62:40105/ using document root <none> and password file <none>
I20260928 06:45:24.416049 10396 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:24.416117 10396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:24.416291 10396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:24.418313 10396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "38ed1c13d20f4de2ac701ba9d5771229"
format_stamp: "Formatted at 2026-09-28 06:45:24 on dist-test-slave-twl6"
server_key: "8d353ec9a3b72782221b652efc44e662"
server_key_iv: "8e1aa6df8366d14f91ee8741565dd8ec"
server_key_version: "encryptionkey@0"
I20260928 06:45:24.418823 10396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "38ed1c13d20f4de2ac701ba9d5771229"
format_stamp: "Formatted at 2026-09-28 06:45:24 on dist-test-slave-twl6"
server_key: "8d353ec9a3b72782221b652efc44e662"
server_key_iv: "8e1aa6df8366d14f91ee8741565dd8ec"
server_key_version: "encryptionkey@0"
I20260928 06:45:24.422637 10396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 06:45:24.425156 10411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:24.426417 10396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:45:24.426550 10396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "38ed1c13d20f4de2ac701ba9d5771229"
format_stamp: "Formatted at 2026-09-28 06:45:24 on dist-test-slave-twl6"
server_key: "8d353ec9a3b72782221b652efc44e662"
server_key_iv: "8e1aa6df8366d14f91ee8741565dd8ec"
server_key_version: "encryptionkey@0"
I20260928 06:45:24.426658 10396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:24.448339 10396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:24.451231 10396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:24.451443 10396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:24.458549 10396 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:40171
I20260928 06:45:24.458846 10463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:40171 every 8 connection(s)
I20260928 06:45:24.459575 10396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:45:24.461755 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 10396
I20260928 06:45:24.461886 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:45:24.462212 7360 external_mini_cluster.cc:1428] Setting key a71f14e3899d0da808314f04d66ecc48
I20260928 06:45:24.462658 10464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:24.468616 10464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38ed1c13d20f4de2ac701ba9d5771229: Bootstrap starting.
Sep 28 06:45:24 dist-test-slave-twl6 krb5kdc[10383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:24.470994 10464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38ed1c13d20f4de2ac701ba9d5771229: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:24.471753 10464 log.cc:824] T 00000000000000000000000000000000 P 38ed1c13d20f4de2ac701ba9d5771229: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:24.473646 10464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38ed1c13d20f4de2ac701ba9d5771229: No bootstrap required, opened a new log
I20260928 06:45:24.476032 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:24.463667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46298 (local address 127.7.48.62:40171)
0928 06:45:24.464107 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:24.464122 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:24.464158 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:45:24.464805 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:24.464821 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:24.465146 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:24.465479 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:24.466294 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:24.467275 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:24.468200 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:24.468685 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:24.468864 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:24.471357 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:24.471362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:24.471374 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:24.471402 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:24.473162 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:24.473614 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:24.473619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:24.473624 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:24.473774 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:24.474059 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:24.474062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:24.474064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:24.474445 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:24.474660 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:24.474932 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 06:45:24.475239 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":153,"threads_started":1}
I20260928 06:45:24.477298 10464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38ed1c13d20f4de2ac701ba9d5771229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ed1c13d20f4de2ac701ba9d5771229" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 40171 } }
I20260928 06:45:24.477504 10464 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38ed1c13d20f4de2ac701ba9d5771229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:24.477558 10464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38ed1c13d20f4de2ac701ba9d5771229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ed1c13d20f4de2ac701ba9d5771229, State: Initialized, Role: FOLLOWER
I20260928 06:45:24.478018 10464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38ed1c13d20f4de2ac701ba9d5771229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ed1c13d20f4de2ac701ba9d5771229" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 40171 } }
I20260928 06:45:24.478178 10464 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38ed1c13d20f4de2ac701ba9d5771229 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:45:24.478250 10464 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38ed1c13d20f4de2ac701ba9d5771229 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:45:24.478335 10464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38ed1c13d20f4de2ac701ba9d5771229 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:24.479137 10464 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38ed1c13d20f4de2ac701ba9d5771229 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ed1c13d20f4de2ac701ba9d5771229" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 40171 } }
I20260928 06:45:24.479413 10464 leader_election.cc:302] T 00000000000000000000000000000000 P 38ed1c13d20f4de2ac701ba9d5771229 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38ed1c13d20f4de2ac701ba9d5771229; no voters:
I20260928 06:45:24.479657 10464 leader_election.cc:288] T 00000000000000000000000000000000 P 38ed1c13d20f4de2ac701ba9d5771229 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:45:24.479848 10469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38ed1c13d20f4de2ac701ba9d5771229 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:24.480125 10469 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38ed1c13d20f4de2ac701ba9d5771229 [term 1 LEADER]: Becoming Leader. State: Replica: 38ed1c13d20f4de2ac701ba9d5771229, State: Running, Role: LEADER
I20260928 06:45:24.480434 10469 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38ed1c13d20f4de2ac701ba9d5771229 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ed1c13d20f4de2ac701ba9d5771229" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 40171 } }
I20260928 06:45:24.480866 10464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38ed1c13d20f4de2ac701ba9d5771229 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:45:24.482106 10471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38ed1c13d20f4de2ac701ba9d5771229 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38ed1c13d20f4de2ac701ba9d5771229. Latest consensus state: current_term: 1 leader_uuid: "38ed1c13d20f4de2ac701ba9d5771229" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ed1c13d20f4de2ac701ba9d5771229" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 40171 } } }
I20260928 06:45:24.482106 10470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38ed1c13d20f4de2ac701ba9d5771229 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38ed1c13d20f4de2ac701ba9d5771229" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ed1c13d20f4de2ac701ba9d5771229" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 40171 } } }
I20260928 06:45:24.482312 10470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38ed1c13d20f4de2ac701ba9d5771229 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:24.482632 10471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38ed1c13d20f4de2ac701ba9d5771229 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:24.482738 10480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:45:24.485831 10480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:45:24.491384 10480 catalog_manager.cc:1347] Generated new cluster ID: 08a857d0bffc4975b01f5bc37484bb83
I20260928 06:45:24.491460 10480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:45:24.505470 10480 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:45:24.506793 10480 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:45:24.517180 10480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38ed1c13d20f4de2ac701ba9d5771229: Generated new TSK 0
I20260928 06:45:24.518072 10480 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
I20260928 06:45:24.555050 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:40171
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:40971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:24.668190 10490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:24.668505 10490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:24.668588 10490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:24.672603 10490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:24.672695 10490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:24.672788 10490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:45:24.678164 10490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:40971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:40171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:24.679312 10490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:24.680413 10490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:24.690323 10495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:24.690328 10498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:24.690348 10496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:24.690905 10490 server_base.cc:1034] running on GCE node
I20260928 06:45:24.691308 10490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:24.691995 10490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:24.693186 10490 hybrid_clock.cc:648] HybridClock initialized: now 1790577924693171 us; error 38 us; skew 500 ppm
Sep 28 06:45:24 dist-test-slave-twl6 krb5kdc[10383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577924, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:24.696012 10490 init.cc:377] Logged in from keytab as kudu/127.7.48.1@KRBTEST.COM (short username kudu)
I20260928 06:45:24.696879 10490 webserver.cc:460] Webserver started at http://127.7.48.1:34193/ using document root <none> and password file <none>
I20260928 06:45:24.697476 10490 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:24.697569 10490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:24.697782 10490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:24.699865 10490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "b8f6e6c187e64c12a5e56f7768898006"
format_stamp: "Formatted at 2026-09-28 06:45:24 on dist-test-slave-twl6"
server_key: "069330ef2e9449f7e198d76179adaf8f"
server_key_iv: "199948c07049e9333c3e326c9fe45fac"
server_key_version: "encryptionkey@0"
I20260928 06:45:24.700495 10490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "b8f6e6c187e64c12a5e56f7768898006"
format_stamp: "Formatted at 2026-09-28 06:45:24 on dist-test-slave-twl6"
server_key: "069330ef2e9449f7e198d76179adaf8f"
server_key_iv: "199948c07049e9333c3e326c9fe45fac"
server_key_version: "encryptionkey@0"
I20260928 06:45:24.704380 10490 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 06:45:24.706832 10505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:24.708148 10490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:45:24.708272 10490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "b8f6e6c187e64c12a5e56f7768898006"
format_stamp: "Formatted at 2026-09-28 06:45:24 on dist-test-slave-twl6"
server_key: "069330ef2e9449f7e198d76179adaf8f"
server_key_iv: "199948c07049e9333c3e326c9fe45fac"
server_key_version: "encryptionkey@0"
I20260928 06:45:24.708381 10490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:24.735548 10490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:24.738639 10490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:24.738854 10490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:24.739476 10490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:24.740381 10490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:24.740449 10490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:24.740530 10490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:24.740574 10490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:24.749732 10490 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:44153
I20260928 06:45:24.749751 10618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:44153 every 8 connection(s)
I20260928 06:45:24.750670 10490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:45:24.751853 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 10490
I20260928 06:45:24.752003 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
I20260928 06:45:24.752354 7360 external_mini_cluster.cc:1428] Setting key 2cb91ac504be63ddcbb2fd4b538785a5
Sep 28 06:45:24 dist-test-slave-twl6 krb5kdc[10383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577924, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.48.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.2@KRBTEST.COM" created.
I20260928 06:45:24.767238 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:24.752563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:46483 (local address 127.7.48.62:40171)
0928 06:45:24.752703 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:24.752708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:24.753659 (+ 951us) server_negotiation.cc:407] Connection header received
0928 06:45:24.755789 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:24.755795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:24.755862 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:24.755994 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:24.757432 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:24.757945 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:24.759797 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:24.760092 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:24.760172 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:24.763239 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:24.763242 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:24.763244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:24.763270 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:24.764660 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:24.765345 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:24.765348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:24.765349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:24.765429 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:24.765889 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:24.765892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:24.765893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:24.766153 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:24.766249 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:24.766804 (+ 555us) server_negotiation.cc:299] Negotiation successful
0928 06:45:24.766959 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:45:24.768172 10621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:24.752999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:40171 (local address 127.7.48.1:46483)
0928 06:45:24.753504 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:24.753548 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:24.755512 (+ 1964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:24.756170 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:24.756177 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:24.756615 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:24.757244 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:24.757267 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:24.758092 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:24.758098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:24.759665 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:24.759681 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:24.760252 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:24.760259 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:24.760499 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:24.761152 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:24.761175 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:24.763057 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:24.764778 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:24.764783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:24.764797 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:24.765149 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:24.765554 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:24.765557 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:24.765558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:24.765782 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:24.766290 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:24.766296 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:24.766555 (+ 259us) client_negotiation.cc:769] Sending connection context
0928 06:45:24.766727 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 06:45:24.767011 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":148,"threads_started":1}
I20260928 06:45:24.769539 10619 heartbeater.cc:347] Connected to a master server at 127.7.48.62:40171
I20260928 06:45:24.769793 10619 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:24.770301 10619 heartbeater.cc:511] Master 127.7.48.62:40171 requested a full tablet report, sending...
I20260928 06:45:24.771927 10428 ts_manager.cc:194] Registered new tserver with Master: b8f6e6c187e64c12a5e56f7768898006 (127.7.48.1:44153)
I20260928 06:45:24.773087 10428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.1@KRBTEST.COM'} at 127.7.48.1:46483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
I20260928 06:45:24.781308 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:40171
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:40971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:24.894080 10624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:24.894347 10624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:24.894392 10624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:24.898494 10624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:24.898552 10624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:24.898628 10624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:45:24.903846 10624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:40971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:40171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:24.905126 10624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:24.906167 10624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:24.916117 10632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:24.916133 10630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:24.916138 10629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:24.916419 10624 server_base.cc:1034] running on GCE node
I20260928 06:45:24.916917 10624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:24.917608 10624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:24.918794 10624 hybrid_clock.cc:648] HybridClock initialized: now 1790577924918761 us; error 46 us; skew 500 ppm
Sep 28 06:45:24 dist-test-slave-twl6 krb5kdc[10383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577924, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:24.921586 10624 init.cc:377] Logged in from keytab as kudu/127.7.48.2@KRBTEST.COM (short username kudu)
I20260928 06:45:24.922508 10624 webserver.cc:460] Webserver started at http://127.7.48.2:43189/ using document root <none> and password file <none>
I20260928 06:45:24.923092 10624 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:24.923178 10624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:24.923405 10624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:24.925531 10624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "afd4969c2bc34691a52a9c6e7edff203"
format_stamp: "Formatted at 2026-09-28 06:45:24 on dist-test-slave-twl6"
server_key: "11c291d5f5914ae66f9c9be7e3127e87"
server_key_iv: "ba170b8e6ff9300c3bc3970bb74fd255"
server_key_version: "encryptionkey@0"
I20260928 06:45:24.926095 10624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "afd4969c2bc34691a52a9c6e7edff203"
format_stamp: "Formatted at 2026-09-28 06:45:24 on dist-test-slave-twl6"
server_key: "11c291d5f5914ae66f9c9be7e3127e87"
server_key_iv: "ba170b8e6ff9300c3bc3970bb74fd255"
server_key_version: "encryptionkey@0"
I20260928 06:45:24.929975 10624 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:45:24.932473 10639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:24.933723 10624 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:45:24.933853 10624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "afd4969c2bc34691a52a9c6e7edff203"
format_stamp: "Formatted at 2026-09-28 06:45:24 on dist-test-slave-twl6"
server_key: "11c291d5f5914ae66f9c9be7e3127e87"
server_key_iv: "ba170b8e6ff9300c3bc3970bb74fd255"
server_key_version: "encryptionkey@0"
I20260928 06:45:24.933960 10624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:24.947654 10624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:24.950965 10624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:24.951177 10624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:24.951771 10624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:24.952625 10624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:24.952690 10624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:24.952757 10624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:24.952803 10624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:24.962051 10624 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:37285
I20260928 06:45:24.962062 10752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:37285 every 8 connection(s)
I20260928 06:45:24.962987 10624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:45:24.968092 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 10624
I20260928 06:45:24.968230 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:45:24.968500 7360 external_mini_cluster.cc:1428] Setting key 3be8bbffdfbb60cc45b6b1cdc93854ad
Sep 28 06:45:24 dist-test-slave-twl6 krb5kdc[10383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577924, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.48.3@KRBTEST.COM; defaulting to no policy
I20260928 06:45:24.978935 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:24.964822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:39859 (local address 127.7.48.62:40171)
0928 06:45:24.964985 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:24.964989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:24.965811 (+ 822us) server_negotiation.cc:407] Connection header received
0928 06:45:24.967225 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:24.967229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:24.967286 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:24.967396 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:24.968823 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:24.969548 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:24.971362 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:24.971599 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:24.971733 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:24.974883 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:24.974886 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:24.974888 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:24.974912 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:24.976496 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:24.977181 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:24.977183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:24.977185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:24.977264 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:24.977669 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:24.977672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:24.977673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:24.977857 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:24.977932 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:24.978642 (+ 710us) server_negotiation.cc:299] Negotiation successful
0928 06:45:24.978783 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 06:45:24.979951 10755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:24.965228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:40171 (local address 127.7.48.2:39859)
0928 06:45:24.965664 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:24.965707 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:24.966982 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:24.967579 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:24.967586 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:24.968037 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:24.968676 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:24.968692 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:24.969674 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:24.969677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:24.971217 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:24.971233 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:24.971796 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:24.971802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:24.972050 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:24.972745 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:24.972763 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:24.974689 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:24.976627 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:24.976643 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:24.976664 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:24.977069 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:24.977359 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:24.977364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:24.977367 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:24.977567 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:24.977966 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:24.977973 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:24.978353 (+ 380us) client_negotiation.cc:769] Sending connection context
0928 06:45:24.978588 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 06:45:24.978998 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.3@KRBTEST.COM" created.
I20260928 06:45:24.981317 10753 heartbeater.cc:347] Connected to a master server at 127.7.48.62:40171
I20260928 06:45:24.981628 10753 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:24.982167 10753 heartbeater.cc:511] Master 127.7.48.62:40171 requested a full tablet report, sending...
I20260928 06:45:24.983316 10428 ts_manager.cc:194] Registered new tserver with Master: afd4969c2bc34691a52a9c6e7edff203 (127.7.48.2:37285)
I20260928 06:45:24.983945 10428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.2@KRBTEST.COM'} at 127.7.48.2:39859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
I20260928 06:45:24.997265 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:40171
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:40971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:25.108928 10758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:25.109197 10758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:25.109239 10758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:25.113320 10758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:25.113384 10758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:25.113489 10758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:45:25.118714 10758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:40971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:40171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:25.119774 10758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:25.120812 10758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:25.130460 10763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:25.130481 10764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:25.130553 10758 server_base.cc:1034] running on GCE node
W20260928 06:45:25.130513 10766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:25.131184 10758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:25.131829 10758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:25.133024 10758 hybrid_clock.cc:648] HybridClock initialized: now 1790577925133009 us; error 26 us; skew 500 ppm
Sep 28 06:45:25 dist-test-slave-twl6 krb5kdc[10383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577925, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:25.135567 10758 init.cc:377] Logged in from keytab as kudu/127.7.48.3@KRBTEST.COM (short username kudu)
I20260928 06:45:25.136420 10758 webserver.cc:460] Webserver started at http://127.7.48.3:34925/ using document root <none> and password file <none>
I20260928 06:45:25.137032 10758 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:25.137128 10758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:25.137349 10758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:25.139482 10758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "a2abcf84e77d483c8e5dbc25ce209f1b"
format_stamp: "Formatted at 2026-09-28 06:45:25 on dist-test-slave-twl6"
server_key: "1c5005f0e754440a0d3b6893955a812e"
server_key_iv: "d9d8695a781d999aa32424733bf1a207"
server_key_version: "encryptionkey@0"
I20260928 06:45:25.140112 10758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "a2abcf84e77d483c8e5dbc25ce209f1b"
format_stamp: "Formatted at 2026-09-28 06:45:25 on dist-test-slave-twl6"
server_key: "1c5005f0e754440a0d3b6893955a812e"
server_key_iv: "d9d8695a781d999aa32424733bf1a207"
server_key_version: "encryptionkey@0"
I20260928 06:45:25.143819 10758 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 06:45:25.146178 10773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:25.147320 10758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 06:45:25.147444 10758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "a2abcf84e77d483c8e5dbc25ce209f1b"
format_stamp: "Formatted at 2026-09-28 06:45:25 on dist-test-slave-twl6"
server_key: "1c5005f0e754440a0d3b6893955a812e"
server_key_iv: "d9d8695a781d999aa32424733bf1a207"
server_key_version: "encryptionkey@0"
I20260928 06:45:25.147550 10758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:25.156744 10758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:25.159983 10758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:25.160175 10758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:25.160739 10758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:25.161553 10758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:25.161597 10758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:25.161660 10758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:25.161696 10758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:25.169855 10758 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:42437
I20260928 06:45:25.169924 10886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:42437 every 8 connection(s)
I20260928 06:45:25.170738 10758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:45:25.173236 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 10758
I20260928 06:45:25.173363 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:45:25.173712 7360 external_mini_cluster.cc:1428] Setting key 367a2fdacd7e6e20271142b9bf70ab04
Sep 28 06:45:25 dist-test-slave-twl6 krb5kdc[10383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577925, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:25.184633 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:25.172649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:35149 (local address 127.7.48.62:40171)
0928 06:45:25.172789 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:25.172793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:25.173716 (+ 923us) server_negotiation.cc:407] Connection header received
0928 06:45:25.175491 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:25.175494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:25.175549 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:25.175635 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:25.176981 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:25.177491 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:25.178595 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:25.178793 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:25.178863 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:25.181268 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:25.181274 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:25.181280 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:25.181310 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:25.182776 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:25.183268 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:25.183271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:25.183272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:25.183345 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:25.183638 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:25.183641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:25.183642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:25.183806 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:25.183947 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:25.184410 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 06:45:25.184510 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:45:25.185448 10889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:25.172988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:40171 (local address 127.7.48.3:35149)
0928 06:45:25.173534 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:25.173596 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:25.175281 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:25.175851 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:25.175861 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:25.176282 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:25.176815 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:25.176830 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:25.177589 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:25.177592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:25.178496 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:25.178504 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:25.178908 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:25.178910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:25.179053 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:25.179567 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:25.179583 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:25.181093 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:25.182878 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:25.182882 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:25.182894 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:25.183176 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:25.183419 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:25.183421 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:25.183423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:25.183561 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:25.183925 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:25.183941 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:25.184183 (+ 242us) client_negotiation.cc:769] Sending connection context
0928 06:45:25.184352 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 06:45:25.184612 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":116,"threads_started":1}
I20260928 06:45:25.186676 10887 heartbeater.cc:347] Connected to a master server at 127.7.48.62:40171
I20260928 06:45:25.186947 10887 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:25.187556 10887 heartbeater.cc:511] Master 127.7.48.62:40171 requested a full tablet report, sending...
I20260928 06:45:25.188628 10428 ts_manager.cc:194] Registered new tserver with Master: a2abcf84e77d483c8e5dbc25ce209f1b (127.7.48.3:42437)
I20260928 06:45:25.189177 10428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.3@KRBTEST.COM'} at 127.7.48.3:35149
I20260928 06:45:25.199263 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:45:25 dist-test-slave-twl6 krb5kdc[10383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577925, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 06:45:25 dist-test-slave-twl6 krb5kdc[10383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577925, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:25.225132 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:25.216274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59544 (local address 127.7.48.62:40171)
0928 06:45:25.216496 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:25.216501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:25.216564 (+ 63us) server_negotiation.cc:407] Connection header received
0928 06:45:25.216754 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:25.216761 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:25.216830 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:25.216942 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:25.217878 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:25.218615 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:25.219490 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:25.219720 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:25.219821 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:25.222081 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:25.222085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:25.222088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:25.222116 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:25.223401 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:25.223834 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:25.223837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:25.223839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:25.223974 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:25.224250 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:25.224254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:25.224256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:25.224475 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:25.224619 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:25.224845 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 06:45:25.224963 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 06:45:25.225613 10428 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59544
I20260928 06:45:25.226428 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 10490
I20260928 06:45:25.232640 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 10624
I20260928 06:45:25.238355 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 10758
I20260928 06:45:25.243551 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 10396
2026-09-28T06:45:25Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2120 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:45:25 dist-test-slave-twl6 krb5kdc[10899](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:45:25 dist-test-slave-twl6 krb5kdc[10899](info): set up 2 sockets
Sep 28 06:45:25 dist-test-slave-twl6 krb5kdc[10899](info): commencing operation
krb5kdc: starting...
W20260928 06:45:26.264523 7360 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:45:26 dist-test-slave-twl6 krb5kdc[10899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:45:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:45:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:45:26.384162 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:42553
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:42553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:26.494371 10912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:26.494694 10912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:26.494761 10912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:26.499111 10912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:45:26.499199 10912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:26.499228 10912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:26.499256 10912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:45:26.499310 10912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:45:26.504810 10912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:42553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:42553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:26.505918 10912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:26.506973 10912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:45:26.515674 10912 server_base.cc:1034] running on GCE node
W20260928 06:45:26.515594 10920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:26.515558 10917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:26.515558 10918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:26.516440 10912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:26.517627 10912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:26.518798 10912 hybrid_clock.cc:648] HybridClock initialized: now 1790577926518764 us; error 52 us; skew 500 ppm
Sep 28 06:45:26 dist-test-slave-twl6 krb5kdc[10899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577926, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:26.522643 10912 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:45:26.523638 10912 webserver.cc:460] Webserver started at http://127.7.48.62:36101/ using document root <none> and password file <none>
I20260928 06:45:26.524394 10912 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:26.524499 10912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:26.524729 10912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:26.526793 10912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "16d1a8a8ca7b4185834cfd44432f23d0"
format_stamp: "Formatted at 2026-09-28 06:45:26 on dist-test-slave-twl6"
server_key: "5ea5375b9d42db3964da02d95a7660ef"
server_key_iv: "95491cb64b41a163939f778d245f057a"
server_key_version: "encryptionkey@0"
I20260928 06:45:26.527647 10912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "16d1a8a8ca7b4185834cfd44432f23d0"
format_stamp: "Formatted at 2026-09-28 06:45:26 on dist-test-slave-twl6"
server_key: "5ea5375b9d42db3964da02d95a7660ef"
server_key_iv: "95491cb64b41a163939f778d245f057a"
server_key_version: "encryptionkey@0"
I20260928 06:45:26.531431 10912 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:45:26.533928 10927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:26.535173 10912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:26.535308 10912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "16d1a8a8ca7b4185834cfd44432f23d0"
format_stamp: "Formatted at 2026-09-28 06:45:26 on dist-test-slave-twl6"
server_key: "5ea5375b9d42db3964da02d95a7660ef"
server_key_iv: "95491cb64b41a163939f778d245f057a"
server_key_version: "encryptionkey@0"
I20260928 06:45:26.535411 10912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:26.561091 10912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:26.564193 10912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:26.564401 10912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:26.571719 10912 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:42553
I20260928 06:45:26.571743 10979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:42553 every 8 connection(s)
I20260928 06:45:26.572759 10912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:45:26.575465 10980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:26.580120 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 10912
I20260928 06:45:26.580250 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:45:26.580540 7360 external_mini_cluster.cc:1428] Setting key 748f1d71b768f1134ef028f3705c4ac5
I20260928 06:45:26.581192 10980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16d1a8a8ca7b4185834cfd44432f23d0: Bootstrap starting.
I20260928 06:45:26.584160 10980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16d1a8a8ca7b4185834cfd44432f23d0: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:26.585026 10980 log.cc:824] T 00000000000000000000000000000000 P 16d1a8a8ca7b4185834cfd44432f23d0: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:26.586968 10980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16d1a8a8ca7b4185834cfd44432f23d0: No bootstrap required, opened a new log
Sep 28 06:45:26 dist-test-slave-twl6 krb5kdc[10899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:26.590807 10980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16d1a8a8ca7b4185834cfd44432f23d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d1a8a8ca7b4185834cfd44432f23d0" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42553 } }
I20260928 06:45:26.591032 10980 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16d1a8a8ca7b4185834cfd44432f23d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:26.591102 10980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16d1a8a8ca7b4185834cfd44432f23d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d1a8a8ca7b4185834cfd44432f23d0, State: Initialized, Role: FOLLOWER
I20260928 06:45:26.591594 10980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16d1a8a8ca7b4185834cfd44432f23d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d1a8a8ca7b4185834cfd44432f23d0" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42553 } }
I20260928 06:45:26.591758 10980 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16d1a8a8ca7b4185834cfd44432f23d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:45:26.591859 10980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16d1a8a8ca7b4185834cfd44432f23d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:45:26.591987 10980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16d1a8a8ca7b4185834cfd44432f23d0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:26.593080 10980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16d1a8a8ca7b4185834cfd44432f23d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d1a8a8ca7b4185834cfd44432f23d0" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42553 } }
I20260928 06:45:26.593446 10980 leader_election.cc:302] T 00000000000000000000000000000000 P 16d1a8a8ca7b4185834cfd44432f23d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16d1a8a8ca7b4185834cfd44432f23d0; no voters:
I20260928 06:45:26.593816 10980 leader_election.cc:288] T 00000000000000000000000000000000 P 16d1a8a8ca7b4185834cfd44432f23d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:45:26.594004 10985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16d1a8a8ca7b4185834cfd44432f23d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:26.594319 10985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16d1a8a8ca7b4185834cfd44432f23d0 [term 1 LEADER]: Becoming Leader. State: Replica: 16d1a8a8ca7b4185834cfd44432f23d0, State: Running, Role: LEADER
I20260928 06:45:26.594702 10985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16d1a8a8ca7b4185834cfd44432f23d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d1a8a8ca7b4185834cfd44432f23d0" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42553 } }
I20260928 06:45:26.595046 10980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16d1a8a8ca7b4185834cfd44432f23d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:45:26.595144 10983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:26.581788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43838 (local address 127.7.48.62:42553)
0928 06:45:26.582160 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:26.582175 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:26.582208 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:45:26.582840 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:26.582856 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:26.583191 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:26.583533 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:26.584411 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:26.585735 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:26.586538 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:26.587189 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:26.587350 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:26.590012 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:26.590016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:26.590029 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:26.590054 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:26.592065 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:26.592461 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:26.592467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:26.592490 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:26.592639 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:26.592897 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:26.592903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:26.592908 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:26.593414 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:26.593627 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:26.593907 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 06:45:26.594211 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":100,"threads_started":1}
I20260928 06:45:26.596024 10986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16d1a8a8ca7b4185834cfd44432f23d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16d1a8a8ca7b4185834cfd44432f23d0. Latest consensus state: current_term: 1 leader_uuid: "16d1a8a8ca7b4185834cfd44432f23d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d1a8a8ca7b4185834cfd44432f23d0" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42553 } } }
I20260928 06:45:26.596155 10986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16d1a8a8ca7b4185834cfd44432f23d0 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:26.596396 10987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16d1a8a8ca7b4185834cfd44432f23d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16d1a8a8ca7b4185834cfd44432f23d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d1a8a8ca7b4185834cfd44432f23d0" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42553 } } }
I20260928 06:45:26.596496 10987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16d1a8a8ca7b4185834cfd44432f23d0 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:26.597453 10989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:45:26.600210 10989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:45:26.606345 10989 catalog_manager.cc:1347] Generated new cluster ID: 57563f29944c4148998e4e46b39dbdc4
I20260928 06:45:26.606419 10989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:45:26.618453 10989 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:45:26.619630 10989 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:45:26.624358 10989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16d1a8a8ca7b4185834cfd44432f23d0: Generated new TSK 0
I20260928 06:45:26.625030 10989 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
I20260928 06:45:26.655499 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:42553
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:26.768333 11006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:26.768601 11006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:26.768644 11006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:26.772743 11006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:26.772800 11006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:26.772861 11006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:45:26.777947 11006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:42553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:26.778980 11006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:26.779991 11006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:26.789357 11011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:26.789346 11012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:26.789453 11006 server_base.cc:1034] running on GCE node
W20260928 06:45:26.789346 11014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:26.789949 11006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:26.790529 11006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:26.791712 11006 hybrid_clock.cc:648] HybridClock initialized: now 1790577926791698 us; error 24 us; skew 500 ppm
Sep 28 06:45:26 dist-test-slave-twl6 krb5kdc[10899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577926, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:26.794457 11006 init.cc:377] Logged in from keytab as kudu/127.7.48.1@KRBTEST.COM (short username kudu)
I20260928 06:45:26.795385 11006 webserver.cc:460] Webserver started at http://127.7.48.1:42505/ using document root <none> and password file <none>
I20260928 06:45:26.795974 11006 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:26.796038 11006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:26.796214 11006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:26.798323 11006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "cee75d4598664e639657ea4d457ebdc9"
format_stamp: "Formatted at 2026-09-28 06:45:26 on dist-test-slave-twl6"
server_key: "45fec8fdfb99fec6b1cfd2c4b0cbb4fa"
server_key_iv: "8f9bfbb1afadba2b15cb0be73fd23788"
server_key_version: "encryptionkey@0"
I20260928 06:45:26.798863 11006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "cee75d4598664e639657ea4d457ebdc9"
format_stamp: "Formatted at 2026-09-28 06:45:26 on dist-test-slave-twl6"
server_key: "45fec8fdfb99fec6b1cfd2c4b0cbb4fa"
server_key_iv: "8f9bfbb1afadba2b15cb0be73fd23788"
server_key_version: "encryptionkey@0"
I20260928 06:45:26.802450 11006 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 06:45:26.804791 11021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:26.805962 11006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:45:26.806074 11006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "cee75d4598664e639657ea4d457ebdc9"
format_stamp: "Formatted at 2026-09-28 06:45:26 on dist-test-slave-twl6"
server_key: "45fec8fdfb99fec6b1cfd2c4b0cbb4fa"
server_key_iv: "8f9bfbb1afadba2b15cb0be73fd23788"
server_key_version: "encryptionkey@0"
I20260928 06:45:26.806190 11006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:26.834271 11006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:26.837574 11006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:26.837787 11006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:26.838478 11006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:26.839423 11006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:26.839485 11006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:26.839550 11006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:26.839582 11006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:26.848493 11006 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:45991
I20260928 06:45:26.848519 11134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:45991 every 8 connection(s)
I20260928 06:45:26.849423 11006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:45:26.852172 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 11006
I20260928 06:45:26.852308 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
I20260928 06:45:26.852617 7360 external_mini_cluster.cc:1428] Setting key 6fd4e2d7d1b3d4ec9be5f8ee9ae19ed0
Sep 28 06:45:26 dist-test-slave-twl6 krb5kdc[10899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577926, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.48.2@KRBTEST.COM; defaulting to no policy
I20260928 06:45:26.864952 10983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:26.851228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:44265 (local address 127.7.48.62:42553)
0928 06:45:26.851407 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:26.851412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:26.852389 (+ 977us) server_negotiation.cc:407] Connection header received
0928 06:45:26.853965 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:26.853969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:26.854019 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:26.854125 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:26.855809 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:26.856716 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:26.857870 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:26.858104 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:26.858224 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:26.861077 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:26.861082 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:26.861085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:26.861118 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:26.862874 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:26.863417 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:26.863421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:26.863423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:26.863509 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:26.863805 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:26.863807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:26.863808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:26.864065 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:26.864168 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:26.864708 (+ 540us) server_negotiation.cc:299] Negotiation successful
0928 06:45:26.864818 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.2@KRBTEST.COM" created.
I20260928 06:45:26.865974 11137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:26.851663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42553 (local address 127.7.48.1:44265)
0928 06:45:26.852206 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:26.852275 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:26.853732 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:26.854317 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:26.854326 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:26.854893 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:26.855614 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:26.855648 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:26.856842 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:26.856845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:26.857739 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:26.857747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:26.858267 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:26.858271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:26.858447 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:26.859119 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:26.859145 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:26.860886 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:26.862986 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:26.862991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:26.863005 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:26.863318 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:26.863597 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:26.863599 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:26.863601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:26.863743 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:26.864177 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:26.864187 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:26.864475 (+ 288us) client_negotiation.cc:769] Sending connection context
0928 06:45:26.864662 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 06:45:26.864931 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":168,"threads_started":1}
I20260928 06:45:26.867316 11135 heartbeater.cc:347] Connected to a master server at 127.7.48.62:42553
I20260928 06:45:26.867580 11135 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:26.868162 11135 heartbeater.cc:511] Master 127.7.48.62:42553 requested a full tablet report, sending...
I20260928 06:45:26.869925 10944 ts_manager.cc:194] Registered new tserver with Master: cee75d4598664e639657ea4d457ebdc9 (127.7.48.1:45991)
I20260928 06:45:26.871704 10944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.1@KRBTEST.COM'} at 127.7.48.1:44265
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
I20260928 06:45:26.881104 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:42553
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:26.990756 11140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:26.991003 11140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:26.991062 11140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:26.995057 11140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:26.995114 11140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:26.995169 11140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:45:27.000605 11140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:42553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:27.001571 11140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:27.002590 11140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:45:27.012295 11140 server_base.cc:1034] running on GCE node
W20260928 06:45:27.012259 11148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:27.012261 11145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:27.012256 11146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:27.012934 11140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:27.013518 11140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:27.014673 11140 hybrid_clock.cc:648] HybridClock initialized: now 1790577927014661 us; error 25 us; skew 500 ppm
Sep 28 06:45:27 dist-test-slave-twl6 krb5kdc[10899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577927, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:27.017274 11140 init.cc:377] Logged in from keytab as kudu/127.7.48.2@KRBTEST.COM (short username kudu)
I20260928 06:45:27.018195 11140 webserver.cc:460] Webserver started at http://127.7.48.2:45429/ using document root <none> and password file <none>
I20260928 06:45:27.018754 11140 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:27.018812 11140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:27.019043 11140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:27.021121 11140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "16e62bbe51c4492293151b1439a7d030"
format_stamp: "Formatted at 2026-09-28 06:45:27 on dist-test-slave-twl6"
server_key: "5f5ee0c796373f19028cd245055057c5"
server_key_iv: "7d78bbe51a190757822df36ab71198b4"
server_key_version: "encryptionkey@0"
I20260928 06:45:27.021667 11140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "16e62bbe51c4492293151b1439a7d030"
format_stamp: "Formatted at 2026-09-28 06:45:27 on dist-test-slave-twl6"
server_key: "5f5ee0c796373f19028cd245055057c5"
server_key_iv: "7d78bbe51a190757822df36ab71198b4"
server_key_version: "encryptionkey@0"
I20260928 06:45:27.025283 11140 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:45:27.027529 11155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:27.028715 11140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:45:27.028837 11140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "16e62bbe51c4492293151b1439a7d030"
format_stamp: "Formatted at 2026-09-28 06:45:27 on dist-test-slave-twl6"
server_key: "5f5ee0c796373f19028cd245055057c5"
server_key_iv: "7d78bbe51a190757822df36ab71198b4"
server_key_version: "encryptionkey@0"
I20260928 06:45:27.028961 11140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:27.039846 11140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:27.042564 11140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:27.042781 11140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:27.043355 11140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:27.044220 11140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:27.044284 11140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:27.044348 11140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:27.044390 11140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:27.053153 11140 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:39203
I20260928 06:45:27.053210 11268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:39203 every 8 connection(s)
I20260928 06:45:27.054091 11140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:45:27.057013 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 11140
I20260928 06:45:27.057143 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:45:27.057485 7360 external_mini_cluster.cc:1428] Setting key 7574caedbc1d153328a6f86f2f7a7def
Sep 28 06:45:27 dist-test-slave-twl6 krb5kdc[10899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577927, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.48.3@KRBTEST.COM; defaulting to no policy
I20260928 06:45:27.069702 10983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:27.055958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:34795 (local address 127.7.48.62:42553)
0928 06:45:27.056108 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:27.056113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:27.057153 (+ 1040us) server_negotiation.cc:407] Connection header received
0928 06:45:27.058705 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:27.058711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:27.058758 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:27.058842 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:27.060437 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:27.061130 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:27.062348 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:27.062788 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:27.062913 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:27.065668 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:27.065675 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:27.065681 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:27.065715 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:27.067498 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:27.068067 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:27.068070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:27.068073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:27.068177 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:27.068503 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:27.068506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:27.068507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:27.068711 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:27.068810 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:27.069374 (+ 564us) server_negotiation.cc:299] Negotiation successful
0928 06:45:27.069516 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.3@KRBTEST.COM" created.
I20260928 06:45:27.070736 11271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:27.056392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42553 (local address 127.7.48.2:34795)
0928 06:45:27.056902 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:27.056962 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:27.058490 (+ 1528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:27.059010 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:27.059017 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:27.059512 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:27.060280 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:27.060301 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:27.061266 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:27.061269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:27.062234 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:27.062243 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:27.062931 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:27.062934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:27.063071 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:27.063730 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:27.063761 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:27.065496 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:27.067625 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:27.067630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:27.067642 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:27.067932 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:27.068278 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:27.068280 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:27.068282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:27.068417 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:27.068810 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:27.068815 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:27.069144 (+ 329us) client_negotiation.cc:769] Sending connection context
0928 06:45:27.069312 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 06:45:27.069574 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":123,"threads_started":1}
I20260928 06:45:27.071964 11269 heartbeater.cc:347] Connected to a master server at 127.7.48.62:42553
I20260928 06:45:27.072238 11269 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:27.072760 11269 heartbeater.cc:511] Master 127.7.48.62:42553 requested a full tablet report, sending...
I20260928 06:45:27.073819 10944 ts_manager.cc:194] Registered new tserver with Master: 16e62bbe51c4492293151b1439a7d030 (127.7.48.2:39203)
I20260928 06:45:27.074394 10944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.2@KRBTEST.COM'} at 127.7.48.2:34795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
I20260928 06:45:27.086661 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:42553
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:27.196986 11274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:27.197273 11274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:27.197346 11274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:27.201442 11274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:27.201524 11274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:27.201614 11274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:45:27.206882 11274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:42553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:27.208045 11274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:27.209219 11274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:27.218961 11280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:27.218977 11279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:27.219045 11274 server_base.cc:1034] running on GCE node
W20260928 06:45:27.218973 11282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:27.219671 11274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:27.220328 11274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:27.221506 11274 hybrid_clock.cc:648] HybridClock initialized: now 1790577927221493 us; error 27 us; skew 500 ppm
Sep 28 06:45:27 dist-test-slave-twl6 krb5kdc[10899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577927, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:27.224668 11274 init.cc:377] Logged in from keytab as kudu/127.7.48.3@KRBTEST.COM (short username kudu)
I20260928 06:45:27.225786 11274 webserver.cc:460] Webserver started at http://127.7.48.3:38851/ using document root <none> and password file <none>
I20260928 06:45:27.226435 11274 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:27.226536 11274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:27.226809 11274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:27.228883 11274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "c16a45e4309d433280af28e21eaecbae"
format_stamp: "Formatted at 2026-09-28 06:45:27 on dist-test-slave-twl6"
server_key: "a057f979d73ad81018f7e6c43d7ca738"
server_key_iv: "32115704c523eb289d2581c103b9304c"
server_key_version: "encryptionkey@0"
I20260928 06:45:27.229426 11274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "c16a45e4309d433280af28e21eaecbae"
format_stamp: "Formatted at 2026-09-28 06:45:27 on dist-test-slave-twl6"
server_key: "a057f979d73ad81018f7e6c43d7ca738"
server_key_iv: "32115704c523eb289d2581c103b9304c"
server_key_version: "encryptionkey@0"
I20260928 06:45:27.233116 11274 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:45:27.235438 11289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:27.236621 11274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 06:45:27.236747 11274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "c16a45e4309d433280af28e21eaecbae"
format_stamp: "Formatted at 2026-09-28 06:45:27 on dist-test-slave-twl6"
server_key: "a057f979d73ad81018f7e6c43d7ca738"
server_key_iv: "32115704c523eb289d2581c103b9304c"
server_key_version: "encryptionkey@0"
I20260928 06:45:27.236856 11274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:27.253335 11274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:27.256079 11274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:27.256271 11274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:27.256855 11274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:27.257664 11274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:27.257737 11274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:27.257808 11274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:27.257855 11274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:27.267172 11274 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:39915
I20260928 06:45:27.267201 11402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:39915 every 8 connection(s)
I20260928 06:45:27.268093 11274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:45:27.272920 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 11274
I20260928 06:45:27.273043 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:45:27.273325 7360 external_mini_cluster.cc:1428] Setting key 8a7dd353fd10f23a32ddccee17568d12
Sep 28 06:45:27 dist-test-slave-twl6 krb5kdc[10899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577927, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:27.281868 10983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:27.269870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:39667 (local address 127.7.48.62:42553)
0928 06:45:27.270017 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:27.270021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:27.270874 (+ 853us) server_negotiation.cc:407] Connection header received
0928 06:45:27.272223 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:27.272227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:27.272276 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:27.272352 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:27.273640 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:27.274338 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:27.275942 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:27.276172 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:27.276251 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:27.278795 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:27.278798 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:27.278800 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:27.278824 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:27.280122 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:27.280608 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:27.280610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:27.280612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:27.280681 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:27.280956 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:27.280958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:27.280959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:27.281119 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:27.281191 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:27.281650 (+ 459us) server_negotiation.cc:299] Negotiation successful
0928 06:45:27.281746 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:45:27.282718 11405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:27.270268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42553 (local address 127.7.48.3:39667)
0928 06:45:27.270716 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:27.270760 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:27.272025 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:27.272513 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:27.272520 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:27.272882 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:27.273480 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:27.273496 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:27.274478 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:27.274482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:27.275795 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:27.275804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:27.276306 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:27.276309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:27.276451 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:27.277108 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:27.277124 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:27.278614 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:27.280215 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:27.280220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:27.280231 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:27.280503 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:27.280754 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:27.280756 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:27.280758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:27.280884 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:27.281199 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:27.281204 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:27.281431 (+ 227us) client_negotiation.cc:769] Sending connection context
0928 06:45:27.281608 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 06:45:27.281850 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":115,"threads_started":1}
I20260928 06:45:27.283842 11403 heartbeater.cc:347] Connected to a master server at 127.7.48.62:42553
I20260928 06:45:27.284114 11403 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:27.284617 11403 heartbeater.cc:511] Master 127.7.48.62:42553 requested a full tablet report, sending...
I20260928 06:45:27.285691 10944 ts_manager.cc:194] Registered new tserver with Master: c16a45e4309d433280af28e21eaecbae (127.7.48.3:39915)
I20260928 06:45:27.286317 10944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.3@KRBTEST.COM'} at 127.7.48.3:39667
I20260928 06:45:27.287364 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:45:27 dist-test-slave-twl6 krb5kdc[10899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:45:27 dist-test-slave-twl6 krb5kdc[10899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.1@KRBTEST.COM
I20260928 06:45:27.312340 11410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:27.303278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58396 (local address 127.7.48.1:45991)
0928 06:45:27.303675 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:27.303686 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:27.303708 (+ 22us) server_negotiation.cc:407] Connection header received
0928 06:45:27.303756 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:27.303762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:27.304023 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:27.304176 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:27.304887 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:27.305621 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:27.306378 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:27.306734 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:27.306813 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:27.309079 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:27.309087 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:27.309100 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:27.309122 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:27.310781 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:27.311160 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:27.311165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:27.311166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:27.311255 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:27.311467 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:27.311469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:27.311470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:27.311715 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:27.311869 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:27.312064 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 06:45:27.312199 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":156,"threads_started":1}
Sep 28 06:45:27 dist-test-slave-twl6 krb5kdc[10899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:27.322808 10983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:27.315303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43850 (local address 127.7.48.62:42553)
0928 06:45:27.315448 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:27.315454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:27.315480 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:45:27.315629 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:27.315632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:27.315680 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:27.315757 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:27.316386 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:27.316947 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:27.317647 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:27.317844 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:27.317918 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:27.319981 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:27.319984 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:27.319985 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:27.320008 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:27.321409 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:27.321750 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:27.321752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:27.321754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:27.321824 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:27.322027 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:27.322030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:27.322032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:27.322223 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:27.322321 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:27.322466 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 06:45:27.322571 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 06:45:27.323192 10944 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43850
I20260928 06:45:27.323745 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 11006
I20260928 06:45:27.330231 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 11140
I20260928 06:45:27.335428 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 11274
I20260928 06:45:27.340796 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 10912
2026-09-28T06:45:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2097 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:45:27 dist-test-slave-twl6 krb5kdc[11415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:45:27 dist-test-slave-twl6 krb5kdc[11415](info): set up 2 sockets
Sep 28 06:45:27 dist-test-slave-twl6 krb5kdc[11415](info): commencing operation
krb5kdc: starting...
W20260928 06:45:28.372879 7360 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:45:28 dist-test-slave-twl6 krb5kdc[11415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:45:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:45:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:45:28.493041 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:39085
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:45933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:39085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:28.606105 11428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:28.606410 11428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:28.606491 11428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:28.610502 11428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:45:28.610599 11428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:28.610636 11428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:28.610673 11428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:45:28.610708 11428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:45:28.615904 11428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:45933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:39085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:39085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:28.616990 11428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:28.618029 11428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:28.626266 11434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:28.626322 11428 server_base.cc:1034] running on GCE node
W20260928 06:45:28.626259 11436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:28.626259 11433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:28.627147 11428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:28.628362 11428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:28.629561 11428 hybrid_clock.cc:648] HybridClock initialized: now 1790577928629553 us; error 38 us; skew 500 ppm
Sep 28 06:45:28 dist-test-slave-twl6 krb5kdc[11415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577928, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:28.635648 11428 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:45:28.636476 11428 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:28.636571 11428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:28.636809 11428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:28.638957 11428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "b34a94031253484f9d1a9614b1c599b2"
format_stamp: "Formatted at 2026-09-28 06:45:28 on dist-test-slave-twl6"
server_key: "742cc1c065d25be01d2de9add9ef959c"
server_key_iv: "5caade69c8a9358cef776f2f55b32bd4"
server_key_version: "encryptionkey@0"
I20260928 06:45:28.639520 11428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "b34a94031253484f9d1a9614b1c599b2"
format_stamp: "Formatted at 2026-09-28 06:45:28 on dist-test-slave-twl6"
server_key: "742cc1c065d25be01d2de9add9ef959c"
server_key_iv: "5caade69c8a9358cef776f2f55b32bd4"
server_key_version: "encryptionkey@0"
I20260928 06:45:28.643343 11428 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260928 06:45:28.645798 11442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:28.647080 11428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:28.647234 11428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "b34a94031253484f9d1a9614b1c599b2"
format_stamp: "Formatted at 2026-09-28 06:45:28 on dist-test-slave-twl6"
server_key: "742cc1c065d25be01d2de9add9ef959c"
server_key_iv: "5caade69c8a9358cef776f2f55b32bd4"
server_key_version: "encryptionkey@0"
I20260928 06:45:28.647351 11428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:28.659605 11428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:28.662557 11428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:28.662783 11428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:28.670092 11428 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:39085
I20260928 06:45:28.670354 11494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:39085 every 8 connection(s)
I20260928 06:45:28.670783 11428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:45:28.673652 11495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:28.679029 11495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34a94031253484f9d1a9614b1c599b2: Bootstrap starting.
I20260928 06:45:28.679370 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 11428
I20260928 06:45:28.679514 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:45:28.679780 7360 external_mini_cluster.cc:1428] Setting key 5e06ebea4ff871ca3707c387f3c5bfb6
I20260928 06:45:28.681687 11495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b34a94031253484f9d1a9614b1c599b2: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:28.682452 11495 log.cc:824] T 00000000000000000000000000000000 P b34a94031253484f9d1a9614b1c599b2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:28.684662 11495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34a94031253484f9d1a9614b1c599b2: No bootstrap required, opened a new log
Sep 28 06:45:28 dist-test-slave-twl6 krb5kdc[11415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:28.689288 11495 raft_consensus.cc:348] T 00000000000000000000000000000000 P b34a94031253484f9d1a9614b1c599b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34a94031253484f9d1a9614b1c599b2" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 39085 } }
I20260928 06:45:28.689523 11495 raft_consensus.cc:374] T 00000000000000000000000000000000 P b34a94031253484f9d1a9614b1c599b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:28.689596 11495 raft_consensus.cc:729] T 00000000000000000000000000000000 P b34a94031253484f9d1a9614b1c599b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b34a94031253484f9d1a9614b1c599b2, State: Initialized, Role: FOLLOWER
I20260928 06:45:28.690122 11495 consensus_queue.cc:260] T 00000000000000000000000000000000 P b34a94031253484f9d1a9614b1c599b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34a94031253484f9d1a9614b1c599b2" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 39085 } }
I20260928 06:45:28.690276 11495 raft_consensus.cc:388] T 00000000000000000000000000000000 P b34a94031253484f9d1a9614b1c599b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:45:28.690375 11495 raft_consensus.cc:482] T 00000000000000000000000000000000 P b34a94031253484f9d1a9614b1c599b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:45:28.690485 11495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b34a94031253484f9d1a9614b1c599b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:28.691766 11495 raft_consensus.cc:504] T 00000000000000000000000000000000 P b34a94031253484f9d1a9614b1c599b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34a94031253484f9d1a9614b1c599b2" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 39085 } }
I20260928 06:45:28.692168 11495 leader_election.cc:302] T 00000000000000000000000000000000 P b34a94031253484f9d1a9614b1c599b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b34a94031253484f9d1a9614b1c599b2; no voters:
I20260928 06:45:28.692554 11495 leader_election.cc:288] T 00000000000000000000000000000000 P b34a94031253484f9d1a9614b1c599b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:45:28.692656 11500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b34a94031253484f9d1a9614b1c599b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:28.692912 11500 raft_consensus.cc:686] T 00000000000000000000000000000000 P b34a94031253484f9d1a9614b1c599b2 [term 1 LEADER]: Becoming Leader. State: Replica: b34a94031253484f9d1a9614b1c599b2, State: Running, Role: LEADER
I20260928 06:45:28.693279 11500 consensus_queue.cc:237] T 00000000000000000000000000000000 P b34a94031253484f9d1a9614b1c599b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34a94031253484f9d1a9614b1c599b2" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 39085 } }
I20260928 06:45:28.694032 11495 sys_catalog.cc:564] T 00000000000000000000000000000000 P b34a94031253484f9d1a9614b1c599b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:45:28.694998 11501 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34a94031253484f9d1a9614b1c599b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b34a94031253484f9d1a9614b1c599b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34a94031253484f9d1a9614b1c599b2" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 39085 } } }
I20260928 06:45:28.695129 11501 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34a94031253484f9d1a9614b1c599b2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:28.695429 11502 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34a94031253484f9d1a9614b1c599b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b34a94031253484f9d1a9614b1c599b2. Latest consensus state: current_term: 1 leader_uuid: "b34a94031253484f9d1a9614b1c599b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34a94031253484f9d1a9614b1c599b2" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 39085 } } }
I20260928 06:45:28.695503 11502 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34a94031253484f9d1a9614b1c599b2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:28.695530 11509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:45:28.695966 11498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:28.681194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51500 (local address 127.7.48.62:39085)
0928 06:45:28.681627 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:28.681649 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:28.681684 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:45:28.682372 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:28.682389 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:28.682773 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:28.683161 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:28.684018 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:28.685063 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:28.686230 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:28.686712 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:28.686826 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:28.689737 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:28.689742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:28.689756 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:28.689784 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:28.691904 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:28.692348 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:28.692354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:28.692360 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:28.692517 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:28.692885 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:28.692888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:28.692890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:28.693400 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:28.694343 (+ 943us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:28.694513 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 06:45:28.694835 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":157,"threads_started":1}
I20260928 06:45:28.698565 11509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:45:28.703984 11509 catalog_manager.cc:1347] Generated new cluster ID: 2e7dbbffa09b482cb07858f8e5e6b26e
I20260928 06:45:28.704053 11509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:45:28.730453 11509 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:45:28.731451 11509 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:45:28.737180 11509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b34a94031253484f9d1a9614b1c599b2: Generated new TSK 0
I20260928 06:45:28.737877 11509 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
I20260928 06:45:28.770536 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:39085
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:45933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:28.884868 11521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:28.885149 11521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:28.885197 11521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:28.889273 11521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:28.889350 11521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:28.889463 11521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:45:28.895195 11521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:45933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:39085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:28.896476 11521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:28.897519 11521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:28.907208 11529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:28.907189 11526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:28.907184 11527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:28.907294 11521 server_base.cc:1034] running on GCE node
I20260928 06:45:28.907994 11521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:28.908676 11521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:28.909946 11521 hybrid_clock.cc:648] HybridClock initialized: now 1790577928909909 us; error 50 us; skew 500 ppm
Sep 28 06:45:28 dist-test-slave-twl6 krb5kdc[11415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577928, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:28.912952 11521 init.cc:377] Logged in from keytab as kudu/127.7.48.1@KRBTEST.COM (short username kudu)
I20260928 06:45:28.913758 11521 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:28.913861 11521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:28.914126 11521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:28.916354 11521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "2f53c4e9f95748c4bb45f172a4d6210a"
format_stamp: "Formatted at 2026-09-28 06:45:28 on dist-test-slave-twl6"
server_key: "f9c472d0ce531d3c793e856b90c3b065"
server_key_iv: "48dc28d9a49423472e734293c60c0fd3"
server_key_version: "encryptionkey@0"
I20260928 06:45:28.916972 11521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "2f53c4e9f95748c4bb45f172a4d6210a"
format_stamp: "Formatted at 2026-09-28 06:45:28 on dist-test-slave-twl6"
server_key: "f9c472d0ce531d3c793e856b90c3b065"
server_key_iv: "48dc28d9a49423472e734293c60c0fd3"
server_key_version: "encryptionkey@0"
I20260928 06:45:28.920861 11521 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 06:45:28.923367 11535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:28.924742 11521 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:45:28.924903 11521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "2f53c4e9f95748c4bb45f172a4d6210a"
format_stamp: "Formatted at 2026-09-28 06:45:28 on dist-test-slave-twl6"
server_key: "f9c472d0ce531d3c793e856b90c3b065"
server_key_iv: "48dc28d9a49423472e734293c60c0fd3"
server_key_version: "encryptionkey@0"
I20260928 06:45:28.925021 11521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:28.939682 11521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:28.943146 11521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:28.943361 11521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:28.944036 11521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:28.944904 11521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:28.944975 11521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:28.945050 11521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:28.945101 11521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:28.954171 11521 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:35069
I20260928 06:45:28.954229 11648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:35069 every 8 connection(s)
I20260928 06:45:28.954823 11521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:45:28.956849 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 11521
I20260928 06:45:28.956984 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
I20260928 06:45:28.957274 7360 external_mini_cluster.cc:1428] Setting key d3ee58fae47937165314af41bae99a4f
Sep 28 06:45:28 dist-test-slave-twl6 krb5kdc[11415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577928, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.48.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.2@KRBTEST.COM" created.
I20260928 06:45:28.970561 11498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:28.956753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:48493 (local address 127.7.48.62:39085)
0928 06:45:28.956939 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:28.956946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:28.957953 (+ 1007us) server_negotiation.cc:407] Connection header received
0928 06:45:28.959812 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:28.959814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:28.959857 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:28.959990 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:28.961396 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:28.962324 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:28.963585 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:28.964077 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:28.964205 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:28.966893 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:28.966897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:28.966900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:28.966941 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:28.968535 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:28.968992 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:28.968994 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:28.968996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:28.969068 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:28.969398 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:28.969402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:28.969404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:28.969602 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:28.969707 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:28.970259 (+ 552us) server_negotiation.cc:299] Negotiation successful
0928 06:45:28.970387 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 06:45:28.971411 11651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:28.957191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:39085 (local address 127.7.48.1:48493)
0928 06:45:28.957765 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:28.957835 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:28.959564 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:28.960215 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:28.960228 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:28.960752 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:28.961243 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:28.961257 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:28.962463 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:28.962465 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:28.963459 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:28.963469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:28.964240 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:28.964242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:28.964388 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:28.965053 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:28.965070 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:28.966716 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:28.968641 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:28.968645 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:28.968659 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:28.968908 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:28.969158 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:28.969160 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:28.969161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:28.969296 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:28.969720 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:28.969730 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:28.969995 (+ 265us) client_negotiation.cc:769] Sending connection context
0928 06:45:28.970217 (+ 222us) client_negotiation.cc:240] Negotiation successful
0928 06:45:28.970572 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":162,"threads_started":1}
I20260928 06:45:28.972905 11649 heartbeater.cc:347] Connected to a master server at 127.7.48.62:39085
I20260928 06:45:28.973169 11649 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:28.973731 11649 heartbeater.cc:511] Master 127.7.48.62:39085 requested a full tablet report, sending...
I20260928 06:45:28.975389 11459 ts_manager.cc:194] Registered new tserver with Master: 2f53c4e9f95748c4bb45f172a4d6210a (127.7.48.1:35069)
I20260928 06:45:28.976589 11459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.1@KRBTEST.COM'} at 127.7.48.1:48493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
I20260928 06:45:28.986797 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:39085
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:45933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:29.103718 11654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:29.104045 11654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:29.104125 11654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:29.108217 11654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:29.108275 11654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:29.108371 11654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:45:29.114347 11654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:45933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:39085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:29.115496 11654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:29.116631 11654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:29.126154 11662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:29.126157 11659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:29.126449 11654 server_base.cc:1034] running on GCE node
W20260928 06:45:29.126157 11660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:29.126940 11654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:29.127552 11654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:29.128734 11654 hybrid_clock.cc:648] HybridClock initialized: now 1790577929128721 us; error 29 us; skew 500 ppm
Sep 28 06:45:29 dist-test-slave-twl6 krb5kdc[11415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577929, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:29.131474 11654 init.cc:377] Logged in from keytab as kudu/127.7.48.2@KRBTEST.COM (short username kudu)
I20260928 06:45:29.132269 11654 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:29.132332 11654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:29.132499 11654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:29.134567 11654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "b2c7ff9cb952405f981d9185c73280fc"
format_stamp: "Formatted at 2026-09-28 06:45:29 on dist-test-slave-twl6"
server_key: "f98ae8714270ddb2f653d289620acd88"
server_key_iv: "7cf5f412c6a34776d5ad5b28e58633bb"
server_key_version: "encryptionkey@0"
I20260928 06:45:29.135074 11654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "b2c7ff9cb952405f981d9185c73280fc"
format_stamp: "Formatted at 2026-09-28 06:45:29 on dist-test-slave-twl6"
server_key: "f98ae8714270ddb2f653d289620acd88"
server_key_iv: "7cf5f412c6a34776d5ad5b28e58633bb"
server_key_version: "encryptionkey@0"
I20260928 06:45:29.138703 11654 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 06:45:29.141208 11668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:29.142457 11654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:45:29.142561 11654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "b2c7ff9cb952405f981d9185c73280fc"
format_stamp: "Formatted at 2026-09-28 06:45:29 on dist-test-slave-twl6"
server_key: "f98ae8714270ddb2f653d289620acd88"
server_key_iv: "7cf5f412c6a34776d5ad5b28e58633bb"
server_key_version: "encryptionkey@0"
I20260928 06:45:29.142648 11654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:29.155694 11654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:29.158623 11654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:29.158782 11654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:29.159312 11654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:29.160221 11654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:29.160283 11654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:29.160324 11654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:29.160341 11654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:29.169750 11654 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:33701
I20260928 06:45:29.169772 11781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:33701 every 8 connection(s)
I20260928 06:45:29.170357 11654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:45:29.173442 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 11654
I20260928 06:45:29.173589 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:45:29.173894 7360 external_mini_cluster.cc:1428] Setting key d3a0c25b685af798dc79f8a34820e7a2
Sep 28 06:45:29 dist-test-slave-twl6 krb5kdc[11415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577929, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.48.3@KRBTEST.COM; defaulting to no policy
I20260928 06:45:29.186329 11498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:29.172071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:52813 (local address 127.7.48.62:39085)
0928 06:45:29.172216 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:29.172219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:29.173069 (+ 850us) server_negotiation.cc:407] Connection header received
0928 06:45:29.174680 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:29.174684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:29.174741 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:29.174831 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:29.176187 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.177054 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.178820 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.179054 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.179139 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:29.182294 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:29.182301 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:29.182306 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:29.182342 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:29.184112 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:29.184747 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:29.184750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:29.184751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:29.184825 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:29.185215 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:29.185217 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:29.185218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:29.185389 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:29.185464 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:29.186082 (+ 618us) server_negotiation.cc:299] Negotiation successful
0928 06:45:29.186197 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.3@KRBTEST.COM" created.
I20260928 06:45:29.187371 11784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:29.172452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:39085 (local address 127.7.48.2:52813)
0928 06:45:29.172916 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:29.172957 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:29.174408 (+ 1451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:29.174983 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:29.174990 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:29.175448 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:29.176039 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:29.176055 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.177225 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.177232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:29.178691 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:29.178705 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.179222 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.179228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:29.179467 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:29.180209 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:29.180236 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:29.182091 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:29.184260 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:29.184265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:29.184278 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:29.184618 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:29.184931 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:29.184934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:29.184935 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:29.185120 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:29.185527 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:29.185553 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:29.185838 (+ 285us) client_negotiation.cc:769] Sending connection context
0928 06:45:29.186004 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 06:45:29.186280 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":120,"threads_started":1}
I20260928 06:45:29.188722 11782 heartbeater.cc:347] Connected to a master server at 127.7.48.62:39085
I20260928 06:45:29.189023 11782 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:29.189600 11782 heartbeater.cc:511] Master 127.7.48.62:39085 requested a full tablet report, sending...
I20260928 06:45:29.190778 11459 ts_manager.cc:194] Registered new tserver with Master: b2c7ff9cb952405f981d9185c73280fc (127.7.48.2:33701)
I20260928 06:45:29.191489 11459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.2@KRBTEST.COM'} at 127.7.48.2:52813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
I20260928 06:45:29.203214 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:39085
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:45933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:29.317621 11787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:29.317929 11787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:29.318008 11787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:29.322109 11787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:29.322196 11787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:29.322294 11787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:45:29.327610 11787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:45933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:39085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:29.328816 11787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:29.329896 11787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:29.339232 11792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:29.339246 11793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:29.339424 11787 server_base.cc:1034] running on GCE node
W20260928 06:45:29.339232 11795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:29.339977 11787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:29.340593 11787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:29.341751 11787 hybrid_clock.cc:648] HybridClock initialized: now 1790577929341744 us; error 39 us; skew 500 ppm
Sep 28 06:45:29 dist-test-slave-twl6 krb5kdc[11415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577929, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:29.344587 11787 init.cc:377] Logged in from keytab as kudu/127.7.48.3@KRBTEST.COM (short username kudu)
I20260928 06:45:29.345270 11787 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:29.345330 11787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:29.345489 11787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:29.347611 11787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "40845c7c6a9d44d59ab4c7dbb84480f0"
format_stamp: "Formatted at 2026-09-28 06:45:29 on dist-test-slave-twl6"
server_key: "f11fdb7046e0a6fb41c4d17990a62e28"
server_key_iv: "6b73f033c388c40a8578f15da98b042c"
server_key_version: "encryptionkey@0"
I20260928 06:45:29.348253 11787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "40845c7c6a9d44d59ab4c7dbb84480f0"
format_stamp: "Formatted at 2026-09-28 06:45:29 on dist-test-slave-twl6"
server_key: "f11fdb7046e0a6fb41c4d17990a62e28"
server_key_iv: "6b73f033c388c40a8578f15da98b042c"
server_key_version: "encryptionkey@0"
I20260928 06:45:29.352135 11787 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 06:45:29.354589 11801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:29.355798 11787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:29.355965 11787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "40845c7c6a9d44d59ab4c7dbb84480f0"
format_stamp: "Formatted at 2026-09-28 06:45:29 on dist-test-slave-twl6"
server_key: "f11fdb7046e0a6fb41c4d17990a62e28"
server_key_iv: "6b73f033c388c40a8578f15da98b042c"
server_key_version: "encryptionkey@0"
I20260928 06:45:29.356104 11787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:29.371757 11787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:29.374744 11787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:29.374953 11787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:29.375545 11787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:29.376457 11787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:29.376539 11787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:29.376636 11787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:29.376688 11787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:29.385670 11787 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:45049
I20260928 06:45:29.385694 11914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:45049 every 8 connection(s)
I20260928 06:45:29.386286 11787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:45:29.390231 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 11787
I20260928 06:45:29.390337 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:45:29.390652 7360 external_mini_cluster.cc:1428] Setting key db35f15a6cca8cd16beefb53ba8c0402
Sep 28 06:45:29 dist-test-slave-twl6 krb5kdc[11415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577929, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:29.400837 11498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:29.388087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:48719 (local address 127.7.48.62:39085)
0928 06:45:29.388229 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:29.388233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:29.389082 (+ 849us) server_negotiation.cc:407] Connection header received
0928 06:45:29.390625 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:29.390628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:29.390694 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:29.390786 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:29.392390 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.393148 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.394390 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.394614 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.394787 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:29.397400 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:29.397403 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:29.397405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:29.397429 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:29.398723 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:29.399252 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:29.399256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:29.399258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:29.399339 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:29.399700 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:29.399703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:29.399705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:29.399938 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:29.400053 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:29.400546 (+ 493us) server_negotiation.cc:299] Negotiation successful
0928 06:45:29.400667 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 06:45:29.401670 11917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:29.388422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:39085 (local address 127.7.48.3:48719)
0928 06:45:29.388903 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:29.388949 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:29.390333 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:29.391008 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:29.391015 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:29.391501 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:29.392238 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:29.392264 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.393262 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.393265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:29.394258 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:29.394267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.394824 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.394827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:29.394966 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:29.395617 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:29.395633 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:29.397211 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:29.398844 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:29.398850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:29.398867 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:29.399136 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:29.399451 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:29.399454 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:29.399456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:29.399598 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:29.400051 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:29.400057 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:29.400301 (+ 244us) client_negotiation.cc:769] Sending connection context
0928 06:45:29.400487 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 06:45:29.400806 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":112,"threads_started":1}
I20260928 06:45:29.402781 11915 heartbeater.cc:347] Connected to a master server at 127.7.48.62:39085
I20260928 06:45:29.403040 11915 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:29.403577 11915 heartbeater.cc:511] Master 127.7.48.62:39085 requested a full tablet report, sending...
I20260928 06:45:29.404798 11458 ts_manager.cc:194] Registered new tserver with Master: 40845c7c6a9d44d59ab4c7dbb84480f0 (127.7.48.3:45049)
I20260928 06:45:29.405359 11458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.3@KRBTEST.COM'} at 127.7.48.3:48719
I20260928 06:45:29.416769 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:45:29.425741 11498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:29.418688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51512 (local address 127.7.48.62:39085)
0928 06:45:29.418834 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:29.418839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:29.419002 (+ 163us) server_negotiation.cc:407] Connection header received
0928 06:45:29.419095 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:29.419097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:29.419143 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:29.419223 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:29.419942 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.420444 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.421233 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.421445 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.421516 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:29.422887 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:29.422890 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:29.422892 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:29.422915 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:29.424212 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:29.424598 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:29.424600 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:29.424601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:29.424673 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:29.424896 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:29.424898 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:29.424899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:29.425112 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:29.425212 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:29.425401 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 06:45:29.425493 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 06:45:29.428763 11458 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:45:29.432001 11458 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:45:29.446882 11927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:29.440313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:33701 (local address 127.0.0.1:53276)
0928 06:45:29.441203 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:29.441255 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:29.441396 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:29.442230 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:29.442236 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:29.442259 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:29.442445 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:29.442452 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.444038 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.444041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:29.445438 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:29.445445 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.446362 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.446365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:29.446449 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:29.446466 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 06:45:29.446515 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:45:29.446733 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":269,"spinlock_wait_cycles":4736,"thread_start_us":122,"threads_started":1}
I20260928 06:45:29.447147 11929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:29.440587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39858 (local address 127.7.48.1:35069)
0928 06:45:29.441733 (+ 1146us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:29.441741 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:29.441759 (+ 18us) server_negotiation.cc:407] Connection header received
0928 06:45:29.441831 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:29.441836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:29.442063 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:29.442221 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:29.442753 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.443517 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.445353 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.446123 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.446232 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:29.446382 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:29.446763 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 06:45:29.446858 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1045,"thread_start_us":78,"threads_started":1}
I20260928 06:45:29.447268 11931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:29.441540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48920 (local address 127.7.48.3:45049)
0928 06:45:29.442011 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:29.442019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:29.442036 (+ 17us) server_negotiation.cc:407] Connection header received
0928 06:45:29.442101 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:29.442105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:29.442300 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:29.442472 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:29.442719 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.443848 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.445175 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.446115 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.446232 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:29.446382 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:29.446595 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 06:45:29.446710 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":170,"threads_started":1}
I20260928 06:45:29.447326 11930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:29.441108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:45049 (local address 127.0.0.1:48920)
0928 06:45:29.441697 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:29.441724 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:29.441805 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:29.442467 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:29.442470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:29.442492 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:29.442592 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:29.442598 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.444038 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.444041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:29.445038 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:29.445048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.446250 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.446255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:29.446382 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:29.446426 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 06:45:29.446540 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 06:45:29.447212 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":612,"spinlock_wait_cycles":2432,"thread_start_us":95,"threads_started":1}
I20260928 06:45:29.447220 11926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:29.440399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:35069 (local address 127.0.0.1:39858)
0928 06:45:29.441201 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:29.441254 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:29.441397 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:29.442434 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:29.442439 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:29.442461 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:29.442591 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:29.442598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.443714 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.443717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:29.445205 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:29.445217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.446469 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.446472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:29.446578 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:29.446603 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 06:45:29.446675 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 06:45:29.447099 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":404,"spinlock_wait_cycles":11904,"thread_start_us":111,"threads_started":1}
I20260928 06:45:29.447763 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:29.440422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53276 (local address 127.7.48.2:33701)
0928 06:45:29.441717 (+ 1295us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:29.441730 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:29.441756 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:45:29.441831 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:29.441836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:29.442063 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:29.442221 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:29.442597 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.443853 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.445553 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.446230 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.447266 (+ 1036us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:29.447442 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:29.447500 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 06:45:29.447602 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1172,"thread_start_us":165,"threads_started":1}
I20260928 06:45:29.450429 11849 tablet_service.cc:1467] Processing CreateTablet for tablet cd8ad29afbe94cc1a30bebc7e4e6b6ca (DEFAULT_TABLE table=test-table [id=9a701638b8ab4e4ca5ccdf5d852e6c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:29.450695 11583 tablet_service.cc:1467] Processing CreateTablet for tablet cd8ad29afbe94cc1a30bebc7e4e6b6ca (DEFAULT_TABLE table=test-table [id=9a701638b8ab4e4ca5ccdf5d852e6c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:29.450693 11716 tablet_service.cc:1467] Processing CreateTablet for tablet cd8ad29afbe94cc1a30bebc7e4e6b6ca (DEFAULT_TABLE table=test-table [id=9a701638b8ab4e4ca5ccdf5d852e6c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:29.451681 11849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd8ad29afbe94cc1a30bebc7e4e6b6ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:29.451732 11583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd8ad29afbe94cc1a30bebc7e4e6b6ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:29.451733 11716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd8ad29afbe94cc1a30bebc7e4e6b6ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:29.458501 11932 tablet_bootstrap.cc:492] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 40845c7c6a9d44d59ab4c7dbb84480f0: Bootstrap starting.
I20260928 06:45:29.459241 11933 tablet_bootstrap.cc:492] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P b2c7ff9cb952405f981d9185c73280fc: Bootstrap starting.
I20260928 06:45:29.459796 11934 tablet_bootstrap.cc:492] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a: Bootstrap starting.
I20260928 06:45:29.461735 11934 tablet_bootstrap.cc:654] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:29.461807 11933 tablet_bootstrap.cc:654] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P b2c7ff9cb952405f981d9185c73280fc: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:29.461817 11932 tablet_bootstrap.cc:654] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 40845c7c6a9d44d59ab4c7dbb84480f0: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:29.463096 11934 log.cc:824] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:29.463191 11932 log.cc:824] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 40845c7c6a9d44d59ab4c7dbb84480f0: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:29.463191 11933 log.cc:824] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P b2c7ff9cb952405f981d9185c73280fc: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:29.465096 11934 tablet_bootstrap.cc:492] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a: No bootstrap required, opened a new log
I20260928 06:45:29.465312 11934 ts_tablet_manager.cc:1397] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 06:45:29.465517 11932 tablet_bootstrap.cc:492] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 40845c7c6a9d44d59ab4c7dbb84480f0: No bootstrap required, opened a new log
I20260928 06:45:29.465514 11933 tablet_bootstrap.cc:492] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P b2c7ff9cb952405f981d9185c73280fc: No bootstrap required, opened a new log
I20260928 06:45:29.465718 11932 ts_tablet_manager.cc:1397] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 40845c7c6a9d44d59ab4c7dbb84480f0: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 06:45:29.465718 11933 ts_tablet_manager.cc:1397] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P b2c7ff9cb952405f981d9185c73280fc: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 06:45:29.468662 11934 raft_consensus.cc:348] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c7ff9cb952405f981d9185c73280fc" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33701 } } peers { permanent_uuid: "2f53c4e9f95748c4bb45f172a4d6210a" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35069 } } peers { permanent_uuid: "40845c7c6a9d44d59ab4c7dbb84480f0" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 45049 } }
I20260928 06:45:29.468920 11934 raft_consensus.cc:374] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:29.468997 11934 raft_consensus.cc:729] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f53c4e9f95748c4bb45f172a4d6210a, State: Initialized, Role: FOLLOWER
I20260928 06:45:29.469509 11934 consensus_queue.cc:260] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c7ff9cb952405f981d9185c73280fc" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33701 } } peers { permanent_uuid: "2f53c4e9f95748c4bb45f172a4d6210a" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35069 } } peers { permanent_uuid: "40845c7c6a9d44d59ab4c7dbb84480f0" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 45049 } }
I20260928 06:45:29.469682 11933 raft_consensus.cc:348] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P b2c7ff9cb952405f981d9185c73280fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c7ff9cb952405f981d9185c73280fc" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33701 } } peers { permanent_uuid: "2f53c4e9f95748c4bb45f172a4d6210a" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35069 } } peers { permanent_uuid: "40845c7c6a9d44d59ab4c7dbb84480f0" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 45049 } }
I20260928 06:45:29.469682 11932 raft_consensus.cc:348] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 40845c7c6a9d44d59ab4c7dbb84480f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c7ff9cb952405f981d9185c73280fc" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33701 } } peers { permanent_uuid: "2f53c4e9f95748c4bb45f172a4d6210a" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35069 } } peers { permanent_uuid: "40845c7c6a9d44d59ab4c7dbb84480f0" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 45049 } }
I20260928 06:45:29.469960 11932 raft_consensus.cc:374] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 40845c7c6a9d44d59ab4c7dbb84480f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:29.470002 11933 raft_consensus.cc:374] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P b2c7ff9cb952405f981d9185c73280fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:29.470100 11933 raft_consensus.cc:729] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P b2c7ff9cb952405f981d9185c73280fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2c7ff9cb952405f981d9185c73280fc, State: Initialized, Role: FOLLOWER
I20260928 06:45:29.470098 11932 raft_consensus.cc:729] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 40845c7c6a9d44d59ab4c7dbb84480f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40845c7c6a9d44d59ab4c7dbb84480f0, State: Initialized, Role: FOLLOWER
I20260928 06:45:29.470571 11933 consensus_queue.cc:260] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P b2c7ff9cb952405f981d9185c73280fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c7ff9cb952405f981d9185c73280fc" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33701 } } peers { permanent_uuid: "2f53c4e9f95748c4bb45f172a4d6210a" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35069 } } peers { permanent_uuid: "40845c7c6a9d44d59ab4c7dbb84480f0" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 45049 } }
I20260928 06:45:29.470577 11932 consensus_queue.cc:260] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 40845c7c6a9d44d59ab4c7dbb84480f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c7ff9cb952405f981d9185c73280fc" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33701 } } peers { permanent_uuid: "2f53c4e9f95748c4bb45f172a4d6210a" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35069 } } peers { permanent_uuid: "40845c7c6a9d44d59ab4c7dbb84480f0" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 45049 } }
I20260928 06:45:29.470758 11934 ts_tablet_manager.cc:1428] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 06:45:29.471122 11649 heartbeater.cc:503] Master 127.7.48.62:39085 was elected leader, sending a full tablet report...
I20260928 06:45:29.471503 11932 ts_tablet_manager.cc:1428] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 40845c7c6a9d44d59ab4c7dbb84480f0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 06:45:29.471503 11933 ts_tablet_manager.cc:1428] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P b2c7ff9cb952405f981d9185c73280fc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 06:45:29.471827 11915 heartbeater.cc:503] Master 127.7.48.62:39085 was elected leader, sending a full tablet report...
I20260928 06:45:29.472434 11782 heartbeater.cc:503] Master 127.7.48.62:39085 was elected leader, sending a full tablet report...
W20260928 06:45:29.637240 11916 tablet.cc:2369] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 40845c7c6a9d44d59ab4c7dbb84480f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:45:29.671416 11783 tablet.cc:2369] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P b2c7ff9cb952405f981d9185c73280fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:45:29.706209 11650 tablet.cc:2369] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:45:29.829666 11938 raft_consensus.cc:482] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:45:29.829879 11938 raft_consensus.cc:504] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c7ff9cb952405f981d9185c73280fc" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33701 } } peers { permanent_uuid: "2f53c4e9f95748c4bb45f172a4d6210a" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35069 } } peers { permanent_uuid: "40845c7c6a9d44d59ab4c7dbb84480f0" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 45049 } }
I20260928 06:45:29.831007 11938 leader_election.cc:288] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2c7ff9cb952405f981d9185c73280fc (127.7.48.2:33701), 40845c7c6a9d44d59ab4c7dbb84480f0 (127.7.48.3:45049)
I20260928 06:45:29.835464 11931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:29.831947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:60299 (local address 127.7.48.3:45049)
0928 06:45:29.832110 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:29.832117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:29.832238 (+ 121us) server_negotiation.cc:407] Connection header received
0928 06:45:29.832363 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:29.832366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:29.832416 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:29.832514 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:29.832774 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.833260 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.834425 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.834843 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.834957 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:29.835025 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:29.835266 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 06:45:29.835333 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 06:45:29.835472 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:29.831406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:56391 (local address 127.7.48.2:33701)
0928 06:45:29.831589 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:29.831596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:29.831841 (+ 245us) server_negotiation.cc:407] Connection header received
0928 06:45:29.832177 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:29.832184 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:29.832250 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:29.832331 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:29.832686 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.833233 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.834431 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.834845 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.834962 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:29.835035 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:29.835270 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 06:45:29.835347 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 06:45:29.835487 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:29.831299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:33701 (local address 127.7.48.1:56391)
0928 06:45:29.831639 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:29.831653 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:29.831951 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:29.832355 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:29.832358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:29.832379 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:29.832560 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:29.832569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.833343 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.833350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:29.834299 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:29.834309 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.834964 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.834968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:29.835104 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:29.835164 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 06:45:29.835251 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:45:29.835334 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":68,"thread_start_us":142,"threads_started":1}
I20260928 06:45:29.835495 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:29.831774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:45049 (local address 127.7.48.1:60299)
0928 06:45:29.832146 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:29.832161 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:29.832266 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:29.832523 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:29.832525 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:29.832539 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:29.832664 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:29.832672 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.833398 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.833401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:29.834301 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:29.834310 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.834966 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.834969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:29.835111 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:29.835167 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 06:45:29.835251 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 06:45:29.835347 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":80,"thread_start_us":136,"threads_started":1}
I20260928 06:45:29.836233 11736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd8ad29afbe94cc1a30bebc7e4e6b6ca" candidate_uuid: "2f53c4e9f95748c4bb45f172a4d6210a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2c7ff9cb952405f981d9185c73280fc" is_pre_election: true
I20260928 06:45:29.836226 11869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd8ad29afbe94cc1a30bebc7e4e6b6ca" candidate_uuid: "2f53c4e9f95748c4bb45f172a4d6210a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40845c7c6a9d44d59ab4c7dbb84480f0" is_pre_election: true
I20260928 06:45:29.836525 11736 raft_consensus.cc:2454] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P b2c7ff9cb952405f981d9185c73280fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f53c4e9f95748c4bb45f172a4d6210a in term 0.
I20260928 06:45:29.836539 11869 raft_consensus.cc:2454] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 40845c7c6a9d44d59ab4c7dbb84480f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f53c4e9f95748c4bb45f172a4d6210a in term 0.
I20260928 06:45:29.836992 11537 leader_election.cc:302] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f53c4e9f95748c4bb45f172a4d6210a, 40845c7c6a9d44d59ab4c7dbb84480f0; no voters:
I20260928 06:45:29.837281 11938 raft_consensus.cc:2781] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:45:29.837355 11938 raft_consensus.cc:482] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:45:29.837384 11938 raft_consensus.cc:3037] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:29.838435 11938 raft_consensus.cc:504] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c7ff9cb952405f981d9185c73280fc" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33701 } } peers { permanent_uuid: "2f53c4e9f95748c4bb45f172a4d6210a" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35069 } } peers { permanent_uuid: "40845c7c6a9d44d59ab4c7dbb84480f0" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 45049 } }
I20260928 06:45:29.838768 11938 leader_election.cc:288] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a [CANDIDATE]: Term 1 election: Requested vote from peers b2c7ff9cb952405f981d9185c73280fc (127.7.48.2:33701), 40845c7c6a9d44d59ab4c7dbb84480f0 (127.7.48.3:45049)
I20260928 06:45:29.839129 11736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd8ad29afbe94cc1a30bebc7e4e6b6ca" candidate_uuid: "2f53c4e9f95748c4bb45f172a4d6210a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2c7ff9cb952405f981d9185c73280fc"
I20260928 06:45:29.839133 11869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd8ad29afbe94cc1a30bebc7e4e6b6ca" candidate_uuid: "2f53c4e9f95748c4bb45f172a4d6210a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40845c7c6a9d44d59ab4c7dbb84480f0"
I20260928 06:45:29.839265 11736 raft_consensus.cc:3037] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P b2c7ff9cb952405f981d9185c73280fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:29.839279 11869 raft_consensus.cc:3037] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 40845c7c6a9d44d59ab4c7dbb84480f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:29.840373 11736 raft_consensus.cc:2454] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P b2c7ff9cb952405f981d9185c73280fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f53c4e9f95748c4bb45f172a4d6210a in term 1.
I20260928 06:45:29.840392 11869 raft_consensus.cc:2454] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 40845c7c6a9d44d59ab4c7dbb84480f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f53c4e9f95748c4bb45f172a4d6210a in term 1.
I20260928 06:45:29.840664 11537 leader_election.cc:302] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f53c4e9f95748c4bb45f172a4d6210a, b2c7ff9cb952405f981d9185c73280fc; no voters:
I20260928 06:45:29.840859 11938 raft_consensus.cc:2781] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:29.841137 11938 raft_consensus.cc:686] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a [term 1 LEADER]: Becoming Leader. State: Replica: 2f53c4e9f95748c4bb45f172a4d6210a, State: Running, Role: LEADER
I20260928 06:45:29.841516 11938 consensus_queue.cc:237] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c7ff9cb952405f981d9185c73280fc" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33701 } } peers { permanent_uuid: "2f53c4e9f95748c4bb45f172a4d6210a" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35069 } } peers { permanent_uuid: "40845c7c6a9d44d59ab4c7dbb84480f0" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 45049 } }
I20260928 06:45:29.844828 11459 catalog_manager.cc:5495] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f53c4e9f95748c4bb45f172a4d6210a (127.7.48.1). New cstate: current_term: 1 leader_uuid: "2f53c4e9f95748c4bb45f172a4d6210a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c7ff9cb952405f981d9185c73280fc" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f53c4e9f95748c4bb45f172a4d6210a" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40845c7c6a9d44d59ab4c7dbb84480f0" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 45049 } health_report { overall_health: UNKNOWN } } }
I20260928 06:45:29.898150 11929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:29.894846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39860 (local address 127.7.48.1:35069)
0928 06:45:29.894968 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:29.894972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:29.894987 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:45:29.895115 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:29.895117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:29.895187 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:29.895335 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:45:29.895599 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.896087 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.896906 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:29.897106 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:29.897189 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:29.897440 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:45:29.897846 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:45:29.897933 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:29.897987 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 06:45:29.898033 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:45:29.904191 11869 raft_consensus.cc:1261] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 40845c7c6a9d44d59ab4c7dbb84480f0 [term 1 FOLLOWER]: Refusing update from remote peer 2f53c4e9f95748c4bb45f172a4d6210a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:29.904286 11736 raft_consensus.cc:1261] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P b2c7ff9cb952405f981d9185c73280fc [term 1 FOLLOWER]: Refusing update from remote peer 2f53c4e9f95748c4bb45f172a4d6210a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:29.904915 11938 consensus_queue.cc:1035] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a [LEADER]: Connected to new peer: Peer: permanent_uuid: "40845c7c6a9d44d59ab4c7dbb84480f0" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 45049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:29.905156 11943 consensus_queue.cc:1035] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 2f53c4e9f95748c4bb45f172a4d6210a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2c7ff9cb952405f981d9185c73280fc" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:29.916446 11946 mvcc.cc:205] Tried to move back new op lower bound from 7334207200879542272 to 7334207200633270272. Current Snapshot: MvccSnapshot[applied={T|T < 7334207200879542272 or (T in {7334207200879542272})}]
I20260928 06:45:29.916685 11948 mvcc.cc:205] Tried to move back new op lower bound from 7334207200879542272 to 7334207200633270272. Current Snapshot: MvccSnapshot[applied={T|T < 7334207200879542272 or (T in {7334207200879542272})}]
I20260928 06:45:29.921836 11459 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:29.921984 11459 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:29.924527 11459 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b34a94031253484f9d1a9614b1c599b2: Sending DeleteTablet for 3 replicas of tablet cd8ad29afbe94cc1a30bebc7e4e6b6ca
I20260928 06:45:29.925431 11583 tablet_service.cc:1514] Processing DeleteTablet for tablet cd8ad29afbe94cc1a30bebc7e4e6b6ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:29 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:39858
I20260928 06:45:29.925652 11716 tablet_service.cc:1514] Processing DeleteTablet for tablet cd8ad29afbe94cc1a30bebc7e4e6b6ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:29 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:53276
I20260928 06:45:29.926183 11956 tablet_replica.cc:330] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P b2c7ff9cb952405f981d9185c73280fc: stopping tablet replica
I20260928 06:45:29.926204 11849 tablet_service.cc:1514] Processing DeleteTablet for tablet cd8ad29afbe94cc1a30bebc7e4e6b6ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:29 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:48920
I20260928 06:45:29.926544 11956 raft_consensus.cc:2229] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P b2c7ff9cb952405f981d9185c73280fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:29.926671 11957 tablet_replica.cc:330] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 40845c7c6a9d44d59ab4c7dbb84480f0: stopping tablet replica
I20260928 06:45:29.926945 11956 raft_consensus.cc:2258] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P b2c7ff9cb952405f981d9185c73280fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:29.927018 11957 raft_consensus.cc:2229] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 40845c7c6a9d44d59ab4c7dbb84480f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:29.927299 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 11521
I20260928 06:45:29.927368 11957 raft_consensus.cc:2258] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 40845c7c6a9d44d59ab4c7dbb84480f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:29.929895 11957 ts_tablet_manager.cc:1905] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 40845c7c6a9d44d59ab4c7dbb84480f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:29.930323 11956 ts_tablet_manager.cc:1905] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P b2c7ff9cb952405f981d9185c73280fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:29.932403 11956 ts_tablet_manager.cc:1918] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P b2c7ff9cb952405f981d9185c73280fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:45:29.932516 11956 log.cc:1196] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P b2c7ff9cb952405f981d9185c73280fc: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/cd8ad29afbe94cc1a30bebc7e4e6b6ca
I20260928 06:45:29.932806 11956 ts_tablet_manager.cc:1939] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P b2c7ff9cb952405f981d9185c73280fc: Deleting consensus metadata
I20260928 06:45:29.933319 11957 ts_tablet_manager.cc:1918] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 40845c7c6a9d44d59ab4c7dbb84480f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:45:29.933419 11957 log.cc:1196] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 40845c7c6a9d44d59ab4c7dbb84480f0: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/cd8ad29afbe94cc1a30bebc7e4e6b6ca
I20260928 06:45:29.933806 11957 ts_tablet_manager.cc:1939] T cd8ad29afbe94cc1a30bebc7e4e6b6ca P 40845c7c6a9d44d59ab4c7dbb84480f0: Deleting consensus metadata
I20260928 06:45:29.934077 11444 catalog_manager.cc:4841] TS b2c7ff9cb952405f981d9185c73280fc (127.7.48.2:33701): tablet cd8ad29afbe94cc1a30bebc7e4e6b6ca (table test-table [id=9a701638b8ab4e4ca5ccdf5d852e6c41]) successfully deleted
I20260928 06:45:29.935005 11444 catalog_manager.cc:4841] TS 40845c7c6a9d44d59ab4c7dbb84480f0 (127.7.48.3:45049): tablet cd8ad29afbe94cc1a30bebc7e4e6b6ca (table test-table [id=9a701638b8ab4e4ca5ccdf5d852e6c41]) successfully deleted
W20260928 06:45:29.935509 11446 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.48.1:35069 (error 108)
I20260928 06:45:29.935740 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 11654
I20260928 06:45:29.937806 11926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:29.937519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:35069 (local address 127.0.0.1:39872)
0928 06:45:29.937644 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:29.937731 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.1:35069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 06:45:29.937983 11446 catalog_manager.cc:4584] TS 2f53c4e9f95748c4bb45f172a4d6210a (127.7.48.1:35069): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cd8ad29afbe94cc1a30bebc7e4e6b6ca: Network error: Client connection negotiation failed: client connection to 127.7.48.1:35069: connect: Connection refused (error 111)
I20260928 06:45:29.941881 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 11787
I20260928 06:45:29.947247 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 11428
2026-09-28T06:45:29Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2606 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T06:45:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:45:29Z Disabled control of system clock
I20260928 06:45:29.977378 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:33277
--webserver_interface=127.7.48.62
--webserver_port=0
--builtin_ntp_servers=127.7.48.20:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:33277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:45:30.103346 11962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:30.103624 11962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:30.103662 11962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:30.107642 11962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:45:30.107703 11962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:30.107718 11962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:30.107735 11962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:45:30.107750 11962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:45:30.112418 11962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:33277
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:33277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:30.113492 11962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:30.114939 11962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:30.123549 11970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:30.123735 11962 server_base.cc:1034] running on GCE node
W20260928 06:45:30.123549 11967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:30.123589 11968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:30.125252 11962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:30.126433 11962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:30.127609 11962 hybrid_clock.cc:648] HybridClock initialized: now 1790577930127593 us; error 28 us; skew 500 ppm
I20260928 06:45:30.129536 11962 webserver.cc:460] Webserver started at http://127.7.48.62:41459/ using document root <none> and password file <none>
I20260928 06:45:30.130134 11962 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:30.130200 11962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:30.130438 11962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:30.132573 11962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "2c638afd2ca544d9b5981a51c2c52a89"
format_stamp: "Formatted at 2026-09-28 06:45:30 on dist-test-slave-twl6"
server_key: "cf57c65233cdb5d2f5733786969cf98d"
server_key_iv: "5fe45e9b40c5feb6564660c0d60ee551"
server_key_version: "encryptionkey@0"
I20260928 06:45:30.133134 11962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "2c638afd2ca544d9b5981a51c2c52a89"
format_stamp: "Formatted at 2026-09-28 06:45:30 on dist-test-slave-twl6"
server_key: "cf57c65233cdb5d2f5733786969cf98d"
server_key_iv: "5fe45e9b40c5feb6564660c0d60ee551"
server_key_version: "encryptionkey@0"
I20260928 06:45:30.136821 11962 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:45:30.139254 11976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:30.140618 11962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:45:30.140758 11962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "2c638afd2ca544d9b5981a51c2c52a89"
format_stamp: "Formatted at 2026-09-28 06:45:30 on dist-test-slave-twl6"
server_key: "cf57c65233cdb5d2f5733786969cf98d"
server_key_iv: "5fe45e9b40c5feb6564660c0d60ee551"
server_key_version: "encryptionkey@0"
I20260928 06:45:30.140872 11962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:30.145586 11962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:30.146193 11962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:30.146373 11962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:30.153267 11962 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:33277
I20260928 06:45:30.153307 12028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:33277 every 8 connection(s)
I20260928 06:45:30.154232 11962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:45:30.157496 12029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:30.162596 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 11962
I20260928 06:45:30.162745 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:45:30.163033 7360 external_mini_cluster.cc:1428] Setting key e57dec7819e79ff8df591dacbcb6d3a7
I20260928 06:45:30.163249 12029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c638afd2ca544d9b5981a51c2c52a89: Bootstrap starting.
I20260928 06:45:30.165834 12029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c638afd2ca544d9b5981a51c2c52a89: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:30.166574 12029 log.cc:824] T 00000000000000000000000000000000 P 2c638afd2ca544d9b5981a51c2c52a89: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:30.168632 12029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c638afd2ca544d9b5981a51c2c52a89: No bootstrap required, opened a new log
I20260928 06:45:30.168672 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:30.164413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43480 (local address 127.7.48.62:33277)
0928 06:45:30.165079 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:30.165094 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:30.165127 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:45:30.165737 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:30.165753 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:30.166156 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:30.166501 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:30.167358 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:30.167363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:30.167374 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:45:30.167383 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:30.167435 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:45:30.167451 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:30.167576 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:30.167742 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 06:45:30.168017 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":140,"threads_started":1}
I20260928 06:45:30.172448 12029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c638afd2ca544d9b5981a51c2c52a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c638afd2ca544d9b5981a51c2c52a89" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33277 } }
I20260928 06:45:30.172679 12029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c638afd2ca544d9b5981a51c2c52a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:30.172758 12029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c638afd2ca544d9b5981a51c2c52a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c638afd2ca544d9b5981a51c2c52a89, State: Initialized, Role: FOLLOWER
I20260928 06:45:30.173170 12029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c638afd2ca544d9b5981a51c2c52a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c638afd2ca544d9b5981a51c2c52a89" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33277 } }
I20260928 06:45:30.173334 12029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c638afd2ca544d9b5981a51c2c52a89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:45:30.173465 12029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c638afd2ca544d9b5981a51c2c52a89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:45:30.173583 12029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c638afd2ca544d9b5981a51c2c52a89 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:30.174458 12029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c638afd2ca544d9b5981a51c2c52a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c638afd2ca544d9b5981a51c2c52a89" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33277 } }
I20260928 06:45:30.174738 12029 leader_election.cc:302] T 00000000000000000000000000000000 P 2c638afd2ca544d9b5981a51c2c52a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c638afd2ca544d9b5981a51c2c52a89; no voters:
I20260928 06:45:30.175004 12029 leader_election.cc:288] T 00000000000000000000000000000000 P 2c638afd2ca544d9b5981a51c2c52a89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:45:30.175122 12034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c638afd2ca544d9b5981a51c2c52a89 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:30.175364 12034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c638afd2ca544d9b5981a51c2c52a89 [term 1 LEADER]: Becoming Leader. State: Replica: 2c638afd2ca544d9b5981a51c2c52a89, State: Running, Role: LEADER
I20260928 06:45:30.175740 12034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c638afd2ca544d9b5981a51c2c52a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c638afd2ca544d9b5981a51c2c52a89" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33277 } }
I20260928 06:45:30.176199 12029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c638afd2ca544d9b5981a51c2c52a89 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:45:30.177356 12036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c638afd2ca544d9b5981a51c2c52a89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c638afd2ca544d9b5981a51c2c52a89. Latest consensus state: current_term: 1 leader_uuid: "2c638afd2ca544d9b5981a51c2c52a89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c638afd2ca544d9b5981a51c2c52a89" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33277 } } }
I20260928 06:45:30.177486 12036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c638afd2ca544d9b5981a51c2c52a89 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:30.177448 12035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c638afd2ca544d9b5981a51c2c52a89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c638afd2ca544d9b5981a51c2c52a89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c638afd2ca544d9b5981a51c2c52a89" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33277 } } }
I20260928 06:45:30.177542 12035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c638afd2ca544d9b5981a51c2c52a89 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:30.177994 12043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:45:30.180778 12043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:45:30.185771 12043 catalog_manager.cc:1347] Generated new cluster ID: 2b7548b2c38a47d994f51d62916f1cb1
I20260928 06:45:30.185830 12043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:45:30.218219 12043 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:45:30.218593 12043 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:45:30.226904 12043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c638afd2ca544d9b5981a51c2c52a89: Generated new TSK 0
I20260928 06:45:30.227708 12043 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:45:30.237743 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:33277
--builtin_ntp_servers=127.7.48.20:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:45:30.351068 12053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:30.351382 12053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:30.351466 12053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:30.355492 12053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:30.355585 12053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:30.355690 12053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:45:30.360556 12053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:33277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:30.361733 12053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:30.363180 12053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:30.372952 12058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:30.372978 12059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:30.373195 12053 server_base.cc:1034] running on GCE node
W20260928 06:45:30.372978 12061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:30.373723 12053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:30.374399 12053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:30.375556 12053 hybrid_clock.cc:648] HybridClock initialized: now 1790577930375542 us; error 27 us; skew 500 ppm
I20260928 06:45:30.377703 12053 webserver.cc:460] Webserver started at http://127.7.48.1:43965/ using document root <none> and password file <none>
I20260928 06:45:30.378290 12053 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:30.378355 12053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:30.378618 12053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:30.380764 12053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "053654f476344fe3b7058be8738f23fb"
format_stamp: "Formatted at 2026-09-28 06:45:30 on dist-test-slave-twl6"
server_key: "6557cba9177d022e17d3f6785091b1f0"
server_key_iv: "fa27af51149091ec8076957fedd8f92c"
server_key_version: "encryptionkey@0"
I20260928 06:45:30.381330 12053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "053654f476344fe3b7058be8738f23fb"
format_stamp: "Formatted at 2026-09-28 06:45:30 on dist-test-slave-twl6"
server_key: "6557cba9177d022e17d3f6785091b1f0"
server_key_iv: "fa27af51149091ec8076957fedd8f92c"
server_key_version: "encryptionkey@0"
I20260928 06:45:30.385049 12053 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:45:30.387595 12067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:30.388898 12053 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 06:45:30.389037 12053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "053654f476344fe3b7058be8738f23fb"
format_stamp: "Formatted at 2026-09-28 06:45:30 on dist-test-slave-twl6"
server_key: "6557cba9177d022e17d3f6785091b1f0"
server_key_iv: "fa27af51149091ec8076957fedd8f92c"
server_key_version: "encryptionkey@0"
I20260928 06:45:30.389153 12053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:30.393895 12053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:30.394488 12053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:30.394678 12053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:30.395217 12053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:30.396173 12053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:30.396242 12053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:30.396311 12053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:30.396361 12053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:30.405182 12053 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:45289
I20260928 06:45:30.405263 12180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:45289 every 8 connection(s)
I20260928 06:45:30.406121 12053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:45:30.412081 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:30.407975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:41129 (local address 127.7.48.62:33277)
0928 06:45:30.408143 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:30.408148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:30.409246 (+ 1098us) server_negotiation.cc:407] Connection header received
0928 06:45:30.410482 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:30.410484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:30.410521 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:30.410598 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:30.411633 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:30.411636 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:30.411638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:45:30.411649 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:30.411668 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:45:30.411676 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:30.411750 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:30.411915 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 06:45:30.411967 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 06:45:30.412984 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:30.408427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33277 (local address 127.7.48.1:41129)
0928 06:45:30.409089 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:30.409130 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:30.410295 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:30.410759 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:30.410765 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:30.410850 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:30.411375 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:45:30.411377 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:30.411415 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:45:30.411477 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:30.411750 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:30.411756 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:30.411760 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:45:30.411931 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 06:45:30.412161 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":117,"threads_started":1}
I20260928 06:45:30.414196 12181 heartbeater.cc:347] Connected to a master server at 127.7.48.62:33277
I20260928 06:45:30.414417 12181 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:30.414456 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 12053
I20260928 06:45:30.414615 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
I20260928 06:45:30.414906 7360 external_mini_cluster.cc:1428] Setting key 4f7de1833d5728043df9dc527abb9bda
I20260928 06:45:30.414917 12181 heartbeater.cc:511] Master 127.7.48.62:33277 requested a full tablet report, sending...
I20260928 06:45:30.416721 11992 ts_manager.cc:194] Registered new tserver with Master: 053654f476344fe3b7058be8738f23fb (127.7.48.1:45289)
I20260928 06:45:30.416932 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:33277
--builtin_ntp_servers=127.7.48.20:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:45:30.532758 12184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:30.533056 12184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:30.533097 12184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:30.537112 12184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:30.537170 12184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:30.537227 12184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:45:30.541932 12184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:33277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:30.543145 12184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:30.544664 12184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:30.554360 12192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:30.554366 12189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:30.554382 12190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:30.554775 12184 server_base.cc:1034] running on GCE node
I20260928 06:45:30.555168 12184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:30.555737 12184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:30.556897 12184 hybrid_clock.cc:648] HybridClock initialized: now 1790577930556890 us; error 37 us; skew 500 ppm
I20260928 06:45:30.558706 12184 webserver.cc:460] Webserver started at http://127.7.48.2:40759/ using document root <none> and password file <none>
I20260928 06:45:30.559255 12184 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:30.559314 12184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:30.559473 12184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:30.561542 12184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "f32ba5013f5d419193513c09fe955b80"
format_stamp: "Formatted at 2026-09-28 06:45:30 on dist-test-slave-twl6"
server_key: "d97ae186954a40668622417430e7692a"
server_key_iv: "e86a9b2990d4c89506ce2c748b860974"
server_key_version: "encryptionkey@0"
I20260928 06:45:30.562053 12184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "f32ba5013f5d419193513c09fe955b80"
format_stamp: "Formatted at 2026-09-28 06:45:30 on dist-test-slave-twl6"
server_key: "d97ae186954a40668622417430e7692a"
server_key_iv: "e86a9b2990d4c89506ce2c748b860974"
server_key_version: "encryptionkey@0"
I20260928 06:45:30.565637 12184 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 06:45:30.567829 12198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:30.569110 12184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:30.569216 12184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "f32ba5013f5d419193513c09fe955b80"
format_stamp: "Formatted at 2026-09-28 06:45:30 on dist-test-slave-twl6"
server_key: "d97ae186954a40668622417430e7692a"
server_key_iv: "e86a9b2990d4c89506ce2c748b860974"
server_key_version: "encryptionkey@0"
I20260928 06:45:30.569320 12184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:30.573913 12184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:30.574465 12184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:30.574623 12184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:30.575114 12184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:30.576048 12184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:30.576097 12184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:30.576134 12184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:30.576156 12184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:30.584815 12184 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:45423
I20260928 06:45:30.584843 12311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:45423 every 8 connection(s)
I20260928 06:45:30.585682 12184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:45:30.591660 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:30.587397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:49709 (local address 127.7.48.62:33277)
0928 06:45:30.587534 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:30.587538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:30.588675 (+ 1137us) server_negotiation.cc:407] Connection header received
0928 06:45:30.589962 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:30.589966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:30.590013 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:30.590130 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:30.591183 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:30.591187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:30.591189 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:45:30.591202 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:30.591227 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:45:30.591236 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:30.591340 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:30.591484 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 06:45:30.591528 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 06:45:30.592608 12314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:30.587815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33277 (local address 127.7.48.2:49709)
0928 06:45:30.588521 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:30.588560 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:30.589751 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:30.590303 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:30.590311 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:30.590399 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:30.590917 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:45:30.590919 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:30.590959 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:45:30.591026 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:30.591334 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:30.591342 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:30.591350 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 06:45:30.591528 (+ 178us) client_negotiation.cc:240] Negotiation successful
0928 06:45:30.591778 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":125,"threads_started":1}
I20260928 06:45:30.593513 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 12184
I20260928 06:45:30.593641 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:45:30.593873 12312 heartbeater.cc:347] Connected to a master server at 127.7.48.62:33277
I20260928 06:45:30.594003 7360 external_mini_cluster.cc:1428] Setting key f350cbacbf606a4cac086b5e1acd4300
I20260928 06:45:30.594110 12312 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:30.594547 12312 heartbeater.cc:511] Master 127.7.48.62:33277 requested a full tablet report, sending...
I20260928 06:45:30.595589 11992 ts_manager.cc:194] Registered new tserver with Master: f32ba5013f5d419193513c09fe955b80 (127.7.48.2:45423)
I20260928 06:45:30.596145 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:33277
--builtin_ntp_servers=127.7.48.20:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:45:30.706180 12315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:30.706449 12315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:30.706487 12315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:30.710515 12315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:30.710582 12315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:30.710683 12315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:45:30.715862 12315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:33277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:30.717062 12315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:30.718519 12315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:30.728016 12321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:30.728152 12315 server_base.cc:1034] running on GCE node
W20260928 06:45:30.728039 12323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:30.728032 12320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:30.728856 12315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:30.729511 12315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:30.730665 12315 hybrid_clock.cc:648] HybridClock initialized: now 1790577930730633 us; error 47 us; skew 500 ppm
I20260928 06:45:30.732481 12315 webserver.cc:460] Webserver started at http://127.7.48.3:35081/ using document root <none> and password file <none>
I20260928 06:45:30.733058 12315 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:30.733136 12315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:30.733387 12315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:30.735419 12315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "d3bc91391996469195b8c2cf83a0ad2f"
format_stamp: "Formatted at 2026-09-28 06:45:30 on dist-test-slave-twl6"
server_key: "b9e7cefbcb7729f00049ee75baf77689"
server_key_iv: "1a33886f4369c5aa3f5e27e4b0d5b009"
server_key_version: "encryptionkey@0"
I20260928 06:45:30.736074 12315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "d3bc91391996469195b8c2cf83a0ad2f"
format_stamp: "Formatted at 2026-09-28 06:45:30 on dist-test-slave-twl6"
server_key: "b9e7cefbcb7729f00049ee75baf77689"
server_key_iv: "1a33886f4369c5aa3f5e27e4b0d5b009"
server_key_version: "encryptionkey@0"
I20260928 06:45:30.739643 12315 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:45:30.742121 12329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:30.743347 12315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:30.743477 12315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "d3bc91391996469195b8c2cf83a0ad2f"
format_stamp: "Formatted at 2026-09-28 06:45:30 on dist-test-slave-twl6"
server_key: "b9e7cefbcb7729f00049ee75baf77689"
server_key_iv: "1a33886f4369c5aa3f5e27e4b0d5b009"
server_key_version: "encryptionkey@0"
I20260928 06:45:30.743602 12315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:30.748359 12315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:30.748911 12315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:30.749045 12315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:30.749533 12315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:30.750383 12315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:30.750432 12315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:30.750469 12315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:30.750530 12315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:30.759186 12315 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:38351
I20260928 06:45:30.759208 12442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:38351 every 8 connection(s)
I20260928 06:45:30.760128 12315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:45:30.762118 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 12315
I20260928 06:45:30.762228 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:45:30.762485 7360 external_mini_cluster.cc:1428] Setting key 93cde4d1e15d03da2a63c45f90dd5ca3
I20260928 06:45:30.766395 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:30.761889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:48497 (local address 127.7.48.62:33277)
0928 06:45:30.762058 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:30.762064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:30.763320 (+ 1256us) server_negotiation.cc:407] Connection header received
0928 06:45:30.764893 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:30.764895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:30.764929 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:30.764993 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:30.766067 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:30.766069 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:30.766071 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:45:30.766079 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:30.766093 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:45:30.766100 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:30.766167 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:30.766267 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 06:45:30.766300 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 06:45:30.767234 12445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:30.762353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33277 (local address 127.7.48.3:48497)
0928 06:45:30.763154 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:30.763209 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:30.764697 (+ 1488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:30.765149 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:30.765155 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:30.765236 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:30.765837 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:45:30.765839 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:30.765877 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:45:30.765941 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:30.766160 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:30.766164 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:30.766167 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:45:30.766282 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 06:45:30.766458 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":137,"threads_started":1}
I20260928 06:45:30.768230 12443 heartbeater.cc:347] Connected to a master server at 127.7.48.62:33277
I20260928 06:45:30.768452 12443 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:30.768868 12443 heartbeater.cc:511] Master 127.7.48.62:33277 requested a full tablet report, sending...
I20260928 06:45:30.769848 11992 ts_manager.cc:194] Registered new tserver with Master: d3bc91391996469195b8c2cf83a0ad2f (127.7.48.3:38351)
I20260928 06:45:30.776238 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:45:30.779619 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:30.778104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43482 (local address 127.7.48.62:33277)
0928 06:45:30.778260 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:30.778265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:30.778439 (+ 174us) server_negotiation.cc:407] Connection header received
0928 06:45:30.778525 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:30.778527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:30.778557 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:30.778618 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:30.779290 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:30.779292 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:30.779294 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:45:30.779301 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:30.779314 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:45:30.779321 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:30.779378 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:30.779421 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 06:45:30.779468 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 06:45:30.782397 11992 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:45:30.785287 11992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:45:30.796730 12454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:30.793599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:45423 (local address 127.0.0.1:50546)
0928 06:45:30.794440 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:30.794455 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:30.794532 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:30.795265 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:30.795270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:30.795291 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:30.795936 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:45:30.795940 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:30.795964 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:45:30.796040 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:30.796380 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:30.796389 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:30.796394 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:45:30.796509 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 06:45:30.796591 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":71,"threads_started":1}
I20260928 06:45:30.796783 12458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:30.794499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50546 (local address 127.7.48.2:45423)
0928 06:45:30.794847 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:30.794860 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:30.794885 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:45:30.794948 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:30.794954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:30.795108 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:30.795555 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:30.796165 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:30.796169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:30.796178 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:45:30.796209 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:30.796253 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:45:30.796279 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:30.796395 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:30.796539 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 06:45:30.796619 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":152,"threads_started":1}
I20260928 06:45:30.796770 12455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:30.793583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:38351 (local address 127.0.0.1:49584)
0928 06:45:30.793992 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:30.794055 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:30.794164 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:30.795187 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:30.795195 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:30.795276 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:30.796137 (+ 861us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:45:30.796139 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:30.796145 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:45:30.796158 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:30.796515 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:30.796518 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:30.796521 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:45:30.796594 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:45:30.796625 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":96,"threads_started":1}
I20260928 06:45:30.797111 12456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:30.793710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49584 (local address 127.7.48.3:38351)
0928 06:45:30.794498 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:30.794512 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:30.794537 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:45:30.794595 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:30.794617 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:30.794821 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:30.794964 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:30.796267 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:30.796273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:30.796281 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:45:30.796293 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:30.796333 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:45:30.796364 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:30.796465 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:30.796874 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 06:45:30.796946 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":158,"threads_started":1}
I20260928 06:45:30.798403 12457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:30.794275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:45289 (local address 127.0.0.1:46810)
0928 06:45:30.795757 (+ 1482us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:30.795771 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:30.795864 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:30.797209 (+ 1345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:30.797212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:30.797225 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:30.797678 (+ 453us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:45:30.797680 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:30.797691 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:45:30.797716 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:30.798186 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:30.798189 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:30.798192 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:45:30.798260 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 06:45:30.798293 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1420,"thread_start_us":106,"threads_started":1}
I20260928 06:45:30.798806 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:30.794720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46810 (local address 127.7.48.1:45289)
0928 06:45:30.795722 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:30.795734 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:30.796231 (+ 497us) server_negotiation.cc:407] Connection header received
0928 06:45:30.796283 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:30.796286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:30.796468 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:30.796594 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:30.797869 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:30.797891 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:30.797905 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:45:30.797919 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:30.797965 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:45:30.798010 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:30.798140 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:30.798555 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 06:45:30.798649 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":853,"thread_start_us":122,"threads_started":1}
I20260928 06:45:30.800088 12246 tablet_service.cc:1467] Processing CreateTablet for tablet 386c348647534099bc35aba40065b26d (DEFAULT_TABLE table=test-table [id=a4c4727ec27d4aa78a61cdba90cd9a96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:30.800314 12377 tablet_service.cc:1467] Processing CreateTablet for tablet 386c348647534099bc35aba40065b26d (DEFAULT_TABLE table=test-table [id=a4c4727ec27d4aa78a61cdba90cd9a96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:30.801270 12246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 386c348647534099bc35aba40065b26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:30.801301 12377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 386c348647534099bc35aba40065b26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:30.802429 12115 tablet_service.cc:1467] Processing CreateTablet for tablet 386c348647534099bc35aba40065b26d (DEFAULT_TABLE table=test-table [id=a4c4727ec27d4aa78a61cdba90cd9a96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:30.803370 12115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 386c348647534099bc35aba40065b26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:30.807302 12460 tablet_bootstrap.cc:492] T 386c348647534099bc35aba40065b26d P d3bc91391996469195b8c2cf83a0ad2f: Bootstrap starting.
I20260928 06:45:30.808578 12461 tablet_bootstrap.cc:492] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80: Bootstrap starting.
I20260928 06:45:30.809973 12460 tablet_bootstrap.cc:654] T 386c348647534099bc35aba40065b26d P d3bc91391996469195b8c2cf83a0ad2f: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:30.810626 12462 tablet_bootstrap.cc:492] T 386c348647534099bc35aba40065b26d P 053654f476344fe3b7058be8738f23fb: Bootstrap starting.
I20260928 06:45:30.810695 12461 tablet_bootstrap.cc:654] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:30.810902 12460 log.cc:824] T 386c348647534099bc35aba40065b26d P d3bc91391996469195b8c2cf83a0ad2f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:30.811529 12461 log.cc:824] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:30.812616 12462 tablet_bootstrap.cc:654] T 386c348647534099bc35aba40065b26d P 053654f476344fe3b7058be8738f23fb: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:30.813069 12460 tablet_bootstrap.cc:492] T 386c348647534099bc35aba40065b26d P d3bc91391996469195b8c2cf83a0ad2f: No bootstrap required, opened a new log
I20260928 06:45:30.813268 12460 ts_tablet_manager.cc:1397] T 386c348647534099bc35aba40065b26d P d3bc91391996469195b8c2cf83a0ad2f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 06:45:30.813362 12462 log.cc:824] T 386c348647534099bc35aba40065b26d P 053654f476344fe3b7058be8738f23fb: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:30.813596 12461 tablet_bootstrap.cc:492] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80: No bootstrap required, opened a new log
I20260928 06:45:30.813781 12461 ts_tablet_manager.cc:1397] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260928 06:45:30.818202 12460 raft_consensus.cc:348] T 386c348647534099bc35aba40065b26d P d3bc91391996469195b8c2cf83a0ad2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bc91391996469195b8c2cf83a0ad2f" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 38351 } } peers { permanent_uuid: "f32ba5013f5d419193513c09fe955b80" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45423 } } peers { permanent_uuid: "053654f476344fe3b7058be8738f23fb" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 45289 } }
I20260928 06:45:30.818279 12461 raft_consensus.cc:348] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bc91391996469195b8c2cf83a0ad2f" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 38351 } } peers { permanent_uuid: "f32ba5013f5d419193513c09fe955b80" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45423 } } peers { permanent_uuid: "053654f476344fe3b7058be8738f23fb" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 45289 } }
I20260928 06:45:30.818508 12461 raft_consensus.cc:374] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:30.818508 12460 raft_consensus.cc:374] T 386c348647534099bc35aba40065b26d P d3bc91391996469195b8c2cf83a0ad2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:30.818562 12461 raft_consensus.cc:729] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f32ba5013f5d419193513c09fe955b80, State: Initialized, Role: FOLLOWER
I20260928 06:45:30.818562 12460 raft_consensus.cc:729] T 386c348647534099bc35aba40065b26d P d3bc91391996469195b8c2cf83a0ad2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3bc91391996469195b8c2cf83a0ad2f, State: Initialized, Role: FOLLOWER
I20260928 06:45:30.819089 12460 consensus_queue.cc:260] T 386c348647534099bc35aba40065b26d P d3bc91391996469195b8c2cf83a0ad2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bc91391996469195b8c2cf83a0ad2f" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 38351 } } peers { permanent_uuid: "f32ba5013f5d419193513c09fe955b80" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45423 } } peers { permanent_uuid: "053654f476344fe3b7058be8738f23fb" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 45289 } }
I20260928 06:45:30.819089 12461 consensus_queue.cc:260] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bc91391996469195b8c2cf83a0ad2f" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 38351 } } peers { permanent_uuid: "f32ba5013f5d419193513c09fe955b80" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45423 } } peers { permanent_uuid: "053654f476344fe3b7058be8738f23fb" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 45289 } }
I20260928 06:45:30.820046 12312 heartbeater.cc:503] Master 127.7.48.62:33277 was elected leader, sending a full tablet report...
I20260928 06:45:30.820080 12443 heartbeater.cc:503] Master 127.7.48.62:33277 was elected leader, sending a full tablet report...
I20260928 06:45:30.820179 12461 ts_tablet_manager.cc:1428] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260928 06:45:30.820564 12462 tablet_bootstrap.cc:492] T 386c348647534099bc35aba40065b26d P 053654f476344fe3b7058be8738f23fb: No bootstrap required, opened a new log
I20260928 06:45:30.820760 12460 ts_tablet_manager.cc:1428] T 386c348647534099bc35aba40065b26d P d3bc91391996469195b8c2cf83a0ad2f: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 06:45:30.820816 12462 ts_tablet_manager.cc:1397] T 386c348647534099bc35aba40065b26d P 053654f476344fe3b7058be8738f23fb: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260928 06:45:30.825558 12462 raft_consensus.cc:348] T 386c348647534099bc35aba40065b26d P 053654f476344fe3b7058be8738f23fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bc91391996469195b8c2cf83a0ad2f" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 38351 } } peers { permanent_uuid: "f32ba5013f5d419193513c09fe955b80" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45423 } } peers { permanent_uuid: "053654f476344fe3b7058be8738f23fb" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 45289 } }
I20260928 06:45:30.825780 12462 raft_consensus.cc:374] T 386c348647534099bc35aba40065b26d P 053654f476344fe3b7058be8738f23fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:30.825845 12462 raft_consensus.cc:729] T 386c348647534099bc35aba40065b26d P 053654f476344fe3b7058be8738f23fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 053654f476344fe3b7058be8738f23fb, State: Initialized, Role: FOLLOWER
I20260928 06:45:30.826251 12462 consensus_queue.cc:260] T 386c348647534099bc35aba40065b26d P 053654f476344fe3b7058be8738f23fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bc91391996469195b8c2cf83a0ad2f" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 38351 } } peers { permanent_uuid: "f32ba5013f5d419193513c09fe955b80" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45423 } } peers { permanent_uuid: "053654f476344fe3b7058be8738f23fb" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 45289 } }
I20260928 06:45:30.826867 12181 heartbeater.cc:503] Master 127.7.48.62:33277 was elected leader, sending a full tablet report...
I20260928 06:45:30.827116 12462 ts_tablet_manager.cc:1428] T 386c348647534099bc35aba40065b26d P 053654f476344fe3b7058be8738f23fb: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
W20260928 06:45:30.836552 12313 tablet.cc:2369] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:45:30.907359 12182 tablet.cc:2369] T 386c348647534099bc35aba40065b26d P 053654f476344fe3b7058be8738f23fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:45:31.011129 12444 tablet.cc:2369] T 386c348647534099bc35aba40065b26d P d3bc91391996469195b8c2cf83a0ad2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:45:31.131915 12466 raft_consensus.cc:482] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:45:31.132155 12466 raft_consensus.cc:504] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bc91391996469195b8c2cf83a0ad2f" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 38351 } } peers { permanent_uuid: "f32ba5013f5d419193513c09fe955b80" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45423 } } peers { permanent_uuid: "053654f476344fe3b7058be8738f23fb" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 45289 } }
I20260928 06:45:31.133378 12466 leader_election.cc:288] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3bc91391996469195b8c2cf83a0ad2f (127.7.48.3:38351), 053654f476344fe3b7058be8738f23fb (127.7.48.1:45289)
I20260928 06:45:31.135638 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:31.133780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:43525 (local address 127.7.48.1:45289)
0928 06:45:31.133924 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:31.133928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:31.134055 (+ 127us) server_negotiation.cc:407] Connection header received
0928 06:45:31.134187 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:31.134193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:31.134243 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:31.134334 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:31.135179 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:31.135182 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:31.135184 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:45:31.135196 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:31.135215 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:45:31.135223 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:31.135324 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:31.135448 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 06:45:31.135496 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 06:45:31.135636 12469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:31.133660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:45289 (local address 127.7.48.2:43525)
0928 06:45:31.133938 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:31.133953 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:31.134074 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:31.134361 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:31.134364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:31.134378 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:31.135003 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:45:31.135004 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:31.135014 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:45:31.135038 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:31.135331 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:31.135334 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:31.135338 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:45:31.135414 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:45:31.135464 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":73,"threads_started":1}
I20260928 06:45:31.135739 12456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:31.133992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:51971 (local address 127.7.48.3:38351)
0928 06:45:31.134121 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:31.134127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:31.134260 (+ 133us) server_negotiation.cc:407] Connection header received
0928 06:45:31.134358 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:31.134360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:31.134398 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:31.134486 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:31.135324 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:31.135328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:31.135331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:45:31.135347 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:31.135373 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:45:31.135398 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:31.135487 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:31.135572 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 06:45:31.135606 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 06:45:31.135766 12470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:31.133660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:38351 (local address 127.7.48.2:51971)
0928 06:45:31.134147 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:31.134162 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:31.134263 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:31.134532 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:31.134535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:31.134666 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:31.135173 (+ 507us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:45:31.135174 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:31.135184 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:45:31.135209 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:31.135482 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:31.135485 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:31.135488 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:45:31.135568 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 06:45:31.135596 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":59,"thread_start_us":98,"threads_started":1}
I20260928 06:45:31.136376 12397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "386c348647534099bc35aba40065b26d" candidate_uuid: "f32ba5013f5d419193513c09fe955b80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3bc91391996469195b8c2cf83a0ad2f" is_pre_election: true
I20260928 06:45:31.136415 12135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "386c348647534099bc35aba40065b26d" candidate_uuid: "f32ba5013f5d419193513c09fe955b80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "053654f476344fe3b7058be8738f23fb" is_pre_election: true
I20260928 06:45:31.136677 12397 raft_consensus.cc:2454] T 386c348647534099bc35aba40065b26d P d3bc91391996469195b8c2cf83a0ad2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f32ba5013f5d419193513c09fe955b80 in term 0.
I20260928 06:45:31.136710 12135 raft_consensus.cc:2454] T 386c348647534099bc35aba40065b26d P 053654f476344fe3b7058be8738f23fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f32ba5013f5d419193513c09fe955b80 in term 0.
I20260928 06:45:31.137171 12202 leader_election.cc:302] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3bc91391996469195b8c2cf83a0ad2f, f32ba5013f5d419193513c09fe955b80; no voters:
I20260928 06:45:31.137470 12466 raft_consensus.cc:2781] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:45:31.137583 12466 raft_consensus.cc:482] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:45:31.137631 12466 raft_consensus.cc:3037] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:31.138675 12466 raft_consensus.cc:504] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bc91391996469195b8c2cf83a0ad2f" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 38351 } } peers { permanent_uuid: "f32ba5013f5d419193513c09fe955b80" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45423 } } peers { permanent_uuid: "053654f476344fe3b7058be8738f23fb" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 45289 } }
I20260928 06:45:31.139016 12466 leader_election.cc:288] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80 [CANDIDATE]: Term 1 election: Requested vote from peers d3bc91391996469195b8c2cf83a0ad2f (127.7.48.3:38351), 053654f476344fe3b7058be8738f23fb (127.7.48.1:45289)
I20260928 06:45:31.139348 12397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "386c348647534099bc35aba40065b26d" candidate_uuid: "f32ba5013f5d419193513c09fe955b80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3bc91391996469195b8c2cf83a0ad2f"
I20260928 06:45:31.139403 12135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "386c348647534099bc35aba40065b26d" candidate_uuid: "f32ba5013f5d419193513c09fe955b80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "053654f476344fe3b7058be8738f23fb"
I20260928 06:45:31.139469 12397 raft_consensus.cc:3037] T 386c348647534099bc35aba40065b26d P d3bc91391996469195b8c2cf83a0ad2f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:31.139551 12135 raft_consensus.cc:3037] T 386c348647534099bc35aba40065b26d P 053654f476344fe3b7058be8738f23fb [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:31.140594 12397 raft_consensus.cc:2454] T 386c348647534099bc35aba40065b26d P d3bc91391996469195b8c2cf83a0ad2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f32ba5013f5d419193513c09fe955b80 in term 1.
I20260928 06:45:31.140723 12135 raft_consensus.cc:2454] T 386c348647534099bc35aba40065b26d P 053654f476344fe3b7058be8738f23fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f32ba5013f5d419193513c09fe955b80 in term 1.
I20260928 06:45:31.140923 12202 leader_election.cc:302] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3bc91391996469195b8c2cf83a0ad2f, f32ba5013f5d419193513c09fe955b80; no voters:
I20260928 06:45:31.141098 12466 raft_consensus.cc:2781] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:31.141300 12466 raft_consensus.cc:686] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80 [term 1 LEADER]: Becoming Leader. State: Replica: f32ba5013f5d419193513c09fe955b80, State: Running, Role: LEADER
I20260928 06:45:31.141688 12466 consensus_queue.cc:237] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bc91391996469195b8c2cf83a0ad2f" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 38351 } } peers { permanent_uuid: "f32ba5013f5d419193513c09fe955b80" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45423 } } peers { permanent_uuid: "053654f476344fe3b7058be8738f23fb" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 45289 } }
I20260928 06:45:31.144901 11993 catalog_manager.cc:5495] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80 reported cstate change: term changed from 0 to 1, leader changed from <none> to f32ba5013f5d419193513c09fe955b80 (127.7.48.2). New cstate: current_term: 1 leader_uuid: "f32ba5013f5d419193513c09fe955b80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bc91391996469195b8c2cf83a0ad2f" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 38351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f32ba5013f5d419193513c09fe955b80" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "053654f476344fe3b7058be8738f23fb" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 45289 } health_report { overall_health: UNKNOWN } } }
I20260928 06:45:31.159729 12458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:31.158055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50550 (local address 127.7.48.2:45423)
0928 06:45:31.158352 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:31.158355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:31.158366 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:45:31.158436 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:31.158438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:31.158493 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:31.158558 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:31.159317 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:31.159320 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:31.159322 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:45:31.159333 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:31.159351 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:45:31.159359 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:31.159472 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:31.159544 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 06:45:31.159578 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205}
I20260928 06:45:31.166074 12397 raft_consensus.cc:1261] T 386c348647534099bc35aba40065b26d P d3bc91391996469195b8c2cf83a0ad2f [term 1 FOLLOWER]: Refusing update from remote peer f32ba5013f5d419193513c09fe955b80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:31.166075 12135 raft_consensus.cc:1261] T 386c348647534099bc35aba40065b26d P 053654f476344fe3b7058be8738f23fb [term 1 FOLLOWER]: Refusing update from remote peer f32ba5013f5d419193513c09fe955b80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:31.166874 12466 consensus_queue.cc:1035] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "053654f476344fe3b7058be8738f23fb" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 45289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:31.167070 12471 consensus_queue.cc:1035] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3bc91391996469195b8c2cf83a0ad2f" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 38351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:31.175284 12476 mvcc.cc:205] Tried to move back new op lower bound from 7334207206048444416 to 7334207205958881280. Current Snapshot: MvccSnapshot[applied={T|T < 7334207206048444416}]
I20260928 06:45:31.175828 12474 mvcc.cc:205] Tried to move back new op lower bound from 7334207206048444416 to 7334207205958881280. Current Snapshot: MvccSnapshot[applied={T|T < 7334207206048444416}]
I20260928 06:45:31.185503 11993 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:31.185711 11993 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:31.188374 11993 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c638afd2ca544d9b5981a51c2c52a89: Sending DeleteTablet for 3 replicas of tablet 386c348647534099bc35aba40065b26d
I20260928 06:45:31.189183 12377 tablet_service.cc:1514] Processing DeleteTablet for tablet 386c348647534099bc35aba40065b26d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:31 UTC) from {username='slave'} at 127.0.0.1:49584
I20260928 06:45:31.189186 12246 tablet_service.cc:1514] Processing DeleteTablet for tablet 386c348647534099bc35aba40065b26d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:31 UTC) from {username='slave'} at 127.0.0.1:50546
I20260928 06:45:31.189255 12115 tablet_service.cc:1514] Processing DeleteTablet for tablet 386c348647534099bc35aba40065b26d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:31 UTC) from {username='slave'} at 127.0.0.1:46810
I20260928 06:45:31.189715 12486 tablet_replica.cc:330] T 386c348647534099bc35aba40065b26d P d3bc91391996469195b8c2cf83a0ad2f: stopping tablet replica
I20260928 06:45:31.190071 12486 raft_consensus.cc:2229] T 386c348647534099bc35aba40065b26d P d3bc91391996469195b8c2cf83a0ad2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:31.190125 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 12053
I20260928 06:45:31.190415 12486 raft_consensus.cc:2258] T 386c348647534099bc35aba40065b26d P d3bc91391996469195b8c2cf83a0ad2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:31.192269 12486 ts_tablet_manager.cc:1905] T 386c348647534099bc35aba40065b26d P d3bc91391996469195b8c2cf83a0ad2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:31.190798 12485 tablet_replica.cc:330] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80: stopping tablet replica
I20260928 06:45:31.193801 12485 raft_consensus.cc:2229] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:45:31.194195 12485 raft_consensus.cc:2258] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:31.195210 12486 ts_tablet_manager.cc:1918] T 386c348647534099bc35aba40065b26d P d3bc91391996469195b8c2cf83a0ad2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:45:31.195375 12486 log.cc:1196] T 386c348647534099bc35aba40065b26d P d3bc91391996469195b8c2cf83a0ad2f: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/386c348647534099bc35aba40065b26d
I20260928 06:45:31.195768 12486 ts_tablet_manager.cc:1939] T 386c348647534099bc35aba40065b26d P d3bc91391996469195b8c2cf83a0ad2f: Deleting consensus metadata
I20260928 06:45:31.195847 12485 ts_tablet_manager.cc:1905] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:31.197095 11980 catalog_manager.cc:4841] TS d3bc91391996469195b8c2cf83a0ad2f (127.7.48.3:38351): tablet 386c348647534099bc35aba40065b26d (table test-table [id=a4c4727ec27d4aa78a61cdba90cd9a96]) successfully deleted
I20260928 06:45:31.199622 12485 ts_tablet_manager.cc:1918] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:45:31.199731 12485 log.cc:1196] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/386c348647534099bc35aba40065b26d
I20260928 06:45:31.200160 12485 ts_tablet_manager.cc:1939] T 386c348647534099bc35aba40065b26d P f32ba5013f5d419193513c09fe955b80: Deleting consensus metadata
W20260928 06:45:31.200201 11978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.48.1:45289 (error 108)
I20260928 06:45:31.201086 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 12184
I20260928 06:45:31.201287 12455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:31.200947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:45289 (local address 127.0.0.1:46820)
0928 06:45:31.201126 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:31.201201 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.1:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260928 06:45:31.202351 11978 catalog_manager.cc:4584] TS 053654f476344fe3b7058be8738f23fb (127.7.48.1:45289): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 386c348647534099bc35aba40065b26d: Network error: Client connection negotiation failed: client connection to 127.7.48.1:45289: connect: Connection refused (error 111)
I20260928 06:45:31.206521 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 12315
I20260928 06:45:31.207530 12455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:31.206687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:45423 (local address 127.0.0.1:50566)
0928 06:45:31.207369 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:31.207432 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.2:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":584}
I20260928 06:45:31.212320 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 11962
2026-09-28T06:45:31Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1264 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 06:45:31.233928 7360 mini_oidc.cc:152] Starting JWKS server
I20260928 06:45:31.235526 7360 webserver.cc:460] Webserver started at https://127.0.0.1:38877/ using document root <none> and password file <none>
I20260928 06:45:31.235716 7360 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 06:45:31.236042 7360 webserver.cc:460] Webserver started at http://127.0.0.1:33645/ using document root <none> and password file <none>
2026-09-28T06:45:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:45:31Z Disabled control of system clock
I20260928 06:45:31.245995 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:34881
--webserver_interface=127.7.48.62
--webserver_port=0
--builtin_ntp_servers=127.7.48.20:39077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:34881
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38877/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577905526996-7360-0/testchainca.cert with env {}
W20260928 06:45:31.358490 12494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:31.358772 12494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:31.358811 12494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:31.362946 12494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:45:31.363009 12494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:31.363031 12494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:31.363049 12494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:45:31.363065 12494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:45:31.363081 12494 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:45:31.363094 12494 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38877/jwks.json
I20260928 06:45:31.368561 12494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:39077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:34881
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:34881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38877/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577905526996-7360-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:31.369661 12494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:31.370718 12494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:31.379297 12500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:31.379372 12502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:31.379302 12499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:31.379472 12494 server_base.cc:1034] running on GCE node
I20260928 06:45:31.380178 12494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:31.381243 12494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:31.382405 12494 hybrid_clock.cc:648] HybridClock initialized: now 1790577931382377 us; error 39 us; skew 500 ppm
I20260928 06:45:31.384284 12494 webserver.cc:460] Webserver started at http://127.7.48.62:36883/ using document root <none> and password file <none>
I20260928 06:45:31.384865 12494 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:31.384927 12494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:31.385154 12494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:31.387159 12494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "db3bc5dce55a4bdaabb5e038290954be"
format_stamp: "Formatted at 2026-09-28 06:45:31 on dist-test-slave-twl6"
server_key: "a19dab4518b0b314b57b61919954d35e"
server_key_iv: "bd60b596119df8d782c6a8e717cdad99"
server_key_version: "encryptionkey@0"
I20260928 06:45:31.387701 12494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "db3bc5dce55a4bdaabb5e038290954be"
format_stamp: "Formatted at 2026-09-28 06:45:31 on dist-test-slave-twl6"
server_key: "a19dab4518b0b314b57b61919954d35e"
server_key_iv: "bd60b596119df8d782c6a8e717cdad99"
server_key_version: "encryptionkey@0"
I20260928 06:45:31.391238 12494 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:45:31.393640 12508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:31.394865 12494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 06:45:31.395000 12494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "db3bc5dce55a4bdaabb5e038290954be"
format_stamp: "Formatted at 2026-09-28 06:45:31 on dist-test-slave-twl6"
server_key: "a19dab4518b0b314b57b61919954d35e"
server_key_iv: "bd60b596119df8d782c6a8e717cdad99"
server_key_version: "encryptionkey@0"
I20260928 06:45:31.395102 12494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:31.424506 12494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:31.425293 12494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:31.425513 12494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:31.432785 12494 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:34881
I20260928 06:45:31.432806 12560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:34881 every 8 connection(s)
I20260928 06:45:31.433789 12494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:45:31.436836 12561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:31.441777 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 12494
I20260928 06:45:31.441901 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:45:31.442193 7360 external_mini_cluster.cc:1428] Setting key 8bb7816f329a993e9f514bbbb37ef974
I20260928 06:45:31.442732 12561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db3bc5dce55a4bdaabb5e038290954be: Bootstrap starting.
I20260928 06:45:31.445389 12561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db3bc5dce55a4bdaabb5e038290954be: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:31.446281 12561 log.cc:824] T 00000000000000000000000000000000 P db3bc5dce55a4bdaabb5e038290954be: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:31.449214 12561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db3bc5dce55a4bdaabb5e038290954be: No bootstrap required, opened a new log
I20260928 06:45:31.452485 12564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:31.443647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46292 (local address 127.7.48.62:34881)
0928 06:45:31.444384 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:31.444403 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:31.444445 (+ 42us) server_negotiation.cc:407] Connection header received
0928 06:45:31.445277 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:31.445304 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:31.445839 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:31.446214 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:31.446676 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:31.447685 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:31.448696 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:31.449369 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:31.449531 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:31.450343 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:31.450349 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:31.450364 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:45:31.450856 (+ 492us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:31.450923 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:45:31.450948 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:31.451050 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:31.451198 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 06:45:31.451468 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":121,"threads_started":1}
I20260928 06:45:31.453665 12561 raft_consensus.cc:348] T 00000000000000000000000000000000 P db3bc5dce55a4bdaabb5e038290954be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db3bc5dce55a4bdaabb5e038290954be" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34881 } }
I20260928 06:45:31.453874 12561 raft_consensus.cc:374] T 00000000000000000000000000000000 P db3bc5dce55a4bdaabb5e038290954be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:31.453941 12561 raft_consensus.cc:729] T 00000000000000000000000000000000 P db3bc5dce55a4bdaabb5e038290954be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db3bc5dce55a4bdaabb5e038290954be, State: Initialized, Role: FOLLOWER
I20260928 06:45:31.454366 12561 consensus_queue.cc:260] T 00000000000000000000000000000000 P db3bc5dce55a4bdaabb5e038290954be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db3bc5dce55a4bdaabb5e038290954be" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34881 } }
I20260928 06:45:31.454516 12561 raft_consensus.cc:388] T 00000000000000000000000000000000 P db3bc5dce55a4bdaabb5e038290954be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:45:31.454609 12561 raft_consensus.cc:482] T 00000000000000000000000000000000 P db3bc5dce55a4bdaabb5e038290954be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:45:31.454736 12561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db3bc5dce55a4bdaabb5e038290954be [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:31.455529 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:45:31.455809 12561 raft_consensus.cc:504] T 00000000000000000000000000000000 P db3bc5dce55a4bdaabb5e038290954be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db3bc5dce55a4bdaabb5e038290954be" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34881 } }
I20260928 06:45:31.456166 12561 leader_election.cc:302] T 00000000000000000000000000000000 P db3bc5dce55a4bdaabb5e038290954be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db3bc5dce55a4bdaabb5e038290954be; no voters:
I20260928 06:45:31.456521 12561 leader_election.cc:288] T 00000000000000000000000000000000 P db3bc5dce55a4bdaabb5e038290954be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:45:31.456619 12566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db3bc5dce55a4bdaabb5e038290954be [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:31.456863 12566 raft_consensus.cc:686] T 00000000000000000000000000000000 P db3bc5dce55a4bdaabb5e038290954be [term 1 LEADER]: Becoming Leader. State: Replica: db3bc5dce55a4bdaabb5e038290954be, State: Running, Role: LEADER
I20260928 06:45:31.457257 12566 consensus_queue.cc:237] T 00000000000000000000000000000000 P db3bc5dce55a4bdaabb5e038290954be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db3bc5dce55a4bdaabb5e038290954be" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34881 } }
I20260928 06:45:31.457827 12561 sys_catalog.cc:564] T 00000000000000000000000000000000 P db3bc5dce55a4bdaabb5e038290954be [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:45:31.458981 12567 sys_catalog.cc:455] T 00000000000000000000000000000000 P db3bc5dce55a4bdaabb5e038290954be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db3bc5dce55a4bdaabb5e038290954be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db3bc5dce55a4bdaabb5e038290954be" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34881 } } }
I20260928 06:45:31.459100 12567 sys_catalog.cc:458] T 00000000000000000000000000000000 P db3bc5dce55a4bdaabb5e038290954be [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:31.459503 12575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:45:31.459460 12568 sys_catalog.cc:455] T 00000000000000000000000000000000 P db3bc5dce55a4bdaabb5e038290954be [sys.catalog]: SysCatalogTable state changed. Reason: New leader db3bc5dce55a4bdaabb5e038290954be. Latest consensus state: current_term: 1 leader_uuid: "db3bc5dce55a4bdaabb5e038290954be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db3bc5dce55a4bdaabb5e038290954be" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34881 } } }
I20260928 06:45:31.460178 12568 sys_catalog.cc:458] T 00000000000000000000000000000000 P db3bc5dce55a4bdaabb5e038290954be [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:31.462798 12575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:45:31.468103 12575 catalog_manager.cc:1347] Generated new cluster ID: 282166aa50004087a891ac5a15295486
I20260928 06:45:31.468168 12575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:45:31.500613 12575 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:45:31.501559 12575 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:45:31.513350 12575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db3bc5dce55a4bdaabb5e038290954be: Generated new TSK 0
I20260928 06:45:31.514258 12575 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 06:45:31.553182 12591 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 06:45:31.553392 12591 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260928 06:45:31.554971 12590 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:31.529363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34881 (local address 127.0.0.1:46304)
0928 06:45:31.529747 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:31.529762 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:31.529898 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:31.530337 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:31.530340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:31.530349 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:45:31.530477 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:31.530483 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:31.531273 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:31.531280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:31.532327 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:31.532338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:31.532788 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:31.532808 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:31.532902 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:31.532957 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:45:31.553650 (+ 20693us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38877/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 06:45:31.553867 (+ 217us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.48.62:34881: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38877/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":120,"threads_started":1}
I20260928 06:45:31.555511 12564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:31.529523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46304 (local address 127.7.48.62:34881)
0928 06:45:31.529658 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:31.529663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:31.529839 (+ 176us) server_negotiation.cc:407] Connection header received
0928 06:45:31.530131 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:31.530134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:31.530198 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:31.530305 (+ 107us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:45:31.530601 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:31.531120 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:31.532446 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:31.532682 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:31.532763 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:31.533163 (+ 400us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:45:31.553323 (+ 20160us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38877/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 06:45:31.555405 (+ 2082us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46304: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38877/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 06:45:31.555625 12564 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46304: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38877/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.7.48.62:34881: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38877/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 06:45:31.556118 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 12494
2026-09-28T06:45:31Z chronyd exiting
I20260928 06:45:31.593825 7360 test_util.cc:175] -----------------------------------------------
I20260928 06:45:31.593968 7360 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577905526996-7360-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (361 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 06:45:31.595196 7360 mini_oidc.cc:152] Starting JWKS server
I20260928 06:45:31.596127 7360 webserver.cc:460] Webserver started at https://127.0.0.1:37559/ using document root <none> and password file <none>
I20260928 06:45:31.596254 7360 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 06:45:31.596529 7360 webserver.cc:460] Webserver started at http://127.0.0.1:42141/ using document root <none> and password file <none>
2026-09-28T06:45:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:45:31Z Disabled control of system clock
I20260928 06:45:31.605630 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:43411
--webserver_interface=127.7.48.62
--webserver_port=0
--builtin_ntp_servers=127.7.48.20:44209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:43411
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37559/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577905526996-7360-0/testchainca.cert with env {}
W20260928 06:45:31.720054 12597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:31.720412 12597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:31.720487 12597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:31.724563 12597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:45:31.724627 12597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:31.724642 12597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:31.724659 12597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:45:31.724675 12597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:45:31.724691 12597 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:45:31.724705 12597 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37559/jwks.json
I20260928 06:45:31.730139 12597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:44209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:43411
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:43411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37559/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577905526996-7360-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:31.731294 12597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:31.732316 12597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:31.741007 12602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:31.741016 12605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:31.741255 12597 server_base.cc:1034] running on GCE node
W20260928 06:45:31.741087 12603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:31.742043 12597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:31.743196 12597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:31.744383 12597 hybrid_clock.cc:648] HybridClock initialized: now 1790577931744370 us; error 30 us; skew 500 ppm
I20260928 06:45:31.746356 12597 webserver.cc:460] Webserver started at http://127.7.48.62:39165/ using document root <none> and password file <none>
I20260928 06:45:31.746973 12597 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:31.747081 12597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:31.747319 12597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:31.749534 12597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "c793067d52414ba98cd3e74de0eca694"
format_stamp: "Formatted at 2026-09-28 06:45:31 on dist-test-slave-twl6"
server_key: "515b8ffeb082d1038247fa70521c64ef"
server_key_iv: "27eb5b55d17895cd22152ce3a9b55877"
server_key_version: "encryptionkey@0"
I20260928 06:45:31.750133 12597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "c793067d52414ba98cd3e74de0eca694"
format_stamp: "Formatted at 2026-09-28 06:45:31 on dist-test-slave-twl6"
server_key: "515b8ffeb082d1038247fa70521c64ef"
server_key_iv: "27eb5b55d17895cd22152ce3a9b55877"
server_key_version: "encryptionkey@0"
I20260928 06:45:31.754021 12597 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 06:45:31.756610 12611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:31.757946 12597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 06:45:31.758113 12597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "c793067d52414ba98cd3e74de0eca694"
format_stamp: "Formatted at 2026-09-28 06:45:31 on dist-test-slave-twl6"
server_key: "515b8ffeb082d1038247fa70521c64ef"
server_key_iv: "27eb5b55d17895cd22152ce3a9b55877"
server_key_version: "encryptionkey@0"
I20260928 06:45:31.758237 12597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:31.768838 12597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:31.769511 12597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:31.769712 12597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:31.776682 12597 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:43411
I20260928 06:45:31.776695 12663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:43411 every 8 connection(s)
I20260928 06:45:31.777650 12597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:45:31.780896 12664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:31.781679 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 12597
I20260928 06:45:31.781834 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:45:31.782156 7360 external_mini_cluster.cc:1428] Setting key 7b71a5d49aa8fb29a86dd05a78364ec5
I20260928 06:45:31.786927 12664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c793067d52414ba98cd3e74de0eca694: Bootstrap starting.
I20260928 06:45:31.789214 12664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c793067d52414ba98cd3e74de0eca694: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:31.789949 12664 log.cc:824] T 00000000000000000000000000000000 P c793067d52414ba98cd3e74de0eca694: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:31.791735 12667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:31.783607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47632 (local address 127.7.48.62:43411)
0928 06:45:31.784298 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:31.784314 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:31.784351 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:45:31.785012 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:31.785028 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:31.785420 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:31.785772 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:31.786201 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:31.787207 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:31.788123 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:31.788604 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:31.788783 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:31.789683 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:31.789688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:31.789701 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:45:31.790147 (+ 446us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:31.790206 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:45:31.790227 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:31.790336 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:31.790475 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 06:45:31.790778 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":165,"threads_started":1}
I20260928 06:45:31.792239 12664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c793067d52414ba98cd3e74de0eca694: No bootstrap required, opened a new log
I20260928 06:45:31.794632 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:45:31.796495 12664 raft_consensus.cc:348] T 00000000000000000000000000000000 P c793067d52414ba98cd3e74de0eca694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c793067d52414ba98cd3e74de0eca694" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 43411 } }
I20260928 06:45:31.796747 12664 raft_consensus.cc:374] T 00000000000000000000000000000000 P c793067d52414ba98cd3e74de0eca694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:31.796826 12664 raft_consensus.cc:729] T 00000000000000000000000000000000 P c793067d52414ba98cd3e74de0eca694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c793067d52414ba98cd3e74de0eca694, State: Initialized, Role: FOLLOWER
I20260928 06:45:31.797276 12664 consensus_queue.cc:260] T 00000000000000000000000000000000 P c793067d52414ba98cd3e74de0eca694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c793067d52414ba98cd3e74de0eca694" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 43411 } }
I20260928 06:45:31.797425 12664 raft_consensus.cc:388] T 00000000000000000000000000000000 P c793067d52414ba98cd3e74de0eca694 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:45:31.797499 12664 raft_consensus.cc:482] T 00000000000000000000000000000000 P c793067d52414ba98cd3e74de0eca694 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:45:31.797632 12664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c793067d52414ba98cd3e74de0eca694 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:31.798597 12664 raft_consensus.cc:504] T 00000000000000000000000000000000 P c793067d52414ba98cd3e74de0eca694 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c793067d52414ba98cd3e74de0eca694" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 43411 } }
I20260928 06:45:31.798954 12664 leader_election.cc:302] T 00000000000000000000000000000000 P c793067d52414ba98cd3e74de0eca694 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c793067d52414ba98cd3e74de0eca694; no voters:
I20260928 06:45:31.799300 12664 leader_election.cc:288] T 00000000000000000000000000000000 P c793067d52414ba98cd3e74de0eca694 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:45:31.799435 12669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c793067d52414ba98cd3e74de0eca694 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:31.799968 12669 raft_consensus.cc:686] T 00000000000000000000000000000000 P c793067d52414ba98cd3e74de0eca694 [term 1 LEADER]: Becoming Leader. State: Replica: c793067d52414ba98cd3e74de0eca694, State: Running, Role: LEADER
I20260928 06:45:31.800351 12669 consensus_queue.cc:237] T 00000000000000000000000000000000 P c793067d52414ba98cd3e74de0eca694 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c793067d52414ba98cd3e74de0eca694" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 43411 } }
I20260928 06:45:31.800765 12664 sys_catalog.cc:564] T 00000000000000000000000000000000 P c793067d52414ba98cd3e74de0eca694 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:45:31.802000 12671 sys_catalog.cc:455] T 00000000000000000000000000000000 P c793067d52414ba98cd3e74de0eca694 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c793067d52414ba98cd3e74de0eca694. Latest consensus state: current_term: 1 leader_uuid: "c793067d52414ba98cd3e74de0eca694" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c793067d52414ba98cd3e74de0eca694" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 43411 } } }
I20260928 06:45:31.802135 12671 sys_catalog.cc:458] T 00000000000000000000000000000000 P c793067d52414ba98cd3e74de0eca694 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:31.802598 12670 sys_catalog.cc:455] T 00000000000000000000000000000000 P c793067d52414ba98cd3e74de0eca694 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c793067d52414ba98cd3e74de0eca694" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c793067d52414ba98cd3e74de0eca694" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 43411 } } }
I20260928 06:45:31.802654 12678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:45:31.804457 12670 sys_catalog.cc:458] T 00000000000000000000000000000000 P c793067d52414ba98cd3e74de0eca694 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:31.805701 12678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:45:31.810957 12678 catalog_manager.cc:1347] Generated new cluster ID: 3bb09122e26342888ed62736e1a84803
I20260928 06:45:31.811049 12678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:45:31.831534 12678 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:45:31.832609 12678 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:45:31.852066 12678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c793067d52414ba98cd3e74de0eca694: Generated new TSK 0
I20260928 06:45:31.852831 12678 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 06:45:31.887455 12694 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 06:45:31.887558 12694 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 06:45:31.888444 12667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:31.866937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47634 (local address 127.7.48.62:43411)
0928 06:45:31.867085 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:31.867091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:31.867310 (+ 219us) server_negotiation.cc:407] Connection header received
0928 06:45:31.867467 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:31.867470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:31.867521 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:31.867622 (+ 101us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:45:31.867947 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:31.868509 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:31.869787 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:31.870012 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:31.870088 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:31.870455 (+ 367us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:45:31.887961 (+ 17506us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 06:45:31.888280 (+ 319us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47634: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 06:45:31.888525 12667 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47634: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 06:45:31.888557 12693 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:31.866763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:43411 (local address 127.0.0.1:47634)
0928 06:45:31.867182 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:31.867198 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:31.867328 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:31.867647 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:31.867651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:31.867659 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:45:31.867783 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:31.867790 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:31.868633 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:31.868637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:31.869649 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:31.869661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:31.870118 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:31.870140 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:31.870243 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:31.870273 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:45:31.888175 (+ 17902us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 06:45:31.888372 (+ 197us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.48.62:43411: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":125,"threads_started":1}
I20260928 06:45:31.889238 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 12597
2026-09-28T06:45:31Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (334 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 06:45:31.929167 7360 mini_oidc.cc:152] Starting JWKS server
I20260928 06:45:31.930073 7360 webserver.cc:460] Webserver started at https://127.0.0.1:33411/ using document root <none> and password file <none>
I20260928 06:45:31.930158 7360 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 06:45:31.930405 7360 webserver.cc:460] Webserver started at http://127.0.0.1:46821/ using document root <none> and password file <none>
2026-09-28T06:45:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:45:31Z Disabled control of system clock
I20260928 06:45:31.939311 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:38141
--webserver_interface=127.7.48.62
--webserver_port=0
--builtin_ntp_servers=127.7.48.20:42563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:38141
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33411/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 06:45:32.051244 12700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:32.051571 12700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:32.051649 12700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:32.055863 12700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:45:32.055961 12700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:32.055989 12700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:32.056027 12700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:45:32.056072 12700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:45:32.056110 12700 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:45:32.056142 12700 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33411/jwks.json
I20260928 06:45:32.060974 12700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:42563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:38141
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:38141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33411/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:32.062144 12700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:32.063591 12700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:32.072177 12706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:32.072283 12700 server_base.cc:1034] running on GCE node
W20260928 06:45:32.072180 12705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:32.072245 12708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:32.072979 12700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:32.074177 12700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:32.075361 12700 hybrid_clock.cc:648] HybridClock initialized: now 1790577932075331 us; error 44 us; skew 500 ppm
I20260928 06:45:32.077242 12700 webserver.cc:460] Webserver started at http://127.7.48.62:35983/ using document root <none> and password file <none>
I20260928 06:45:32.077843 12700 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:32.077946 12700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:32.078190 12700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:32.080322 12700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "543cea5c4e5849ab876e36f510ccb56a"
format_stamp: "Formatted at 2026-09-28 06:45:32 on dist-test-slave-twl6"
server_key: "8ff7f5df1cf323df8b7ab92b03e85611"
server_key_iv: "c2219984bc596a98b4ee6dbe593ad39f"
server_key_version: "encryptionkey@0"
I20260928 06:45:32.080888 12700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "543cea5c4e5849ab876e36f510ccb56a"
format_stamp: "Formatted at 2026-09-28 06:45:32 on dist-test-slave-twl6"
server_key: "8ff7f5df1cf323df8b7ab92b03e85611"
server_key_iv: "c2219984bc596a98b4ee6dbe593ad39f"
server_key_version: "encryptionkey@0"
I20260928 06:45:32.084673 12700 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 06:45:32.087078 12714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:32.088344 12700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 06:45:32.088482 12700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "543cea5c4e5849ab876e36f510ccb56a"
format_stamp: "Formatted at 2026-09-28 06:45:32 on dist-test-slave-twl6"
server_key: "8ff7f5df1cf323df8b7ab92b03e85611"
server_key_iv: "c2219984bc596a98b4ee6dbe593ad39f"
server_key_version: "encryptionkey@0"
I20260928 06:45:32.088603 12700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:32.103473 12700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:32.104223 12700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:32.104456 12700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:32.111716 12700 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:38141
I20260928 06:45:32.111739 12766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:38141 every 8 connection(s)
I20260928 06:45:32.112717 12700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:45:32.114709 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 12700
I20260928 06:45:32.114806 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:45:32.115103 7360 external_mini_cluster.cc:1428] Setting key a5dddff536d909f5a150930129c27c3b
I20260928 06:45:32.116575 12767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:32.122884 12767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 543cea5c4e5849ab876e36f510ccb56a: Bootstrap starting.
I20260928 06:45:32.124219 12770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:32.116410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52158 (local address 127.7.48.62:38141)
0928 06:45:32.117015 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:32.117031 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:32.117067 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:45:32.117713 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:32.117730 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:32.118113 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:32.118403 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:32.118770 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:32.119728 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:32.120936 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:32.121393 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:32.121508 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:32.122415 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:32.122420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:32.122432 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:45:32.122819 (+ 387us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:32.122874 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:45:32.122893 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:32.122966 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:32.123173 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 06:45:32.123451 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":132,"threads_started":1}
I20260928 06:45:32.125238 12767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 543cea5c4e5849ab876e36f510ccb56a: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:32.125955 12767 log.cc:824] T 00000000000000000000000000000000 P 543cea5c4e5849ab876e36f510ccb56a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:32.126932 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:45:32.128221 12767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 543cea5c4e5849ab876e36f510ccb56a: No bootstrap required, opened a new log
I20260928 06:45:32.131665 12767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 543cea5c4e5849ab876e36f510ccb56a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543cea5c4e5849ab876e36f510ccb56a" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 38141 } }
I20260928 06:45:32.131916 12767 raft_consensus.cc:374] T 00000000000000000000000000000000 P 543cea5c4e5849ab876e36f510ccb56a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:32.131985 12767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 543cea5c4e5849ab876e36f510ccb56a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 543cea5c4e5849ab876e36f510ccb56a, State: Initialized, Role: FOLLOWER
I20260928 06:45:32.132439 12767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 543cea5c4e5849ab876e36f510ccb56a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543cea5c4e5849ab876e36f510ccb56a" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 38141 } }
I20260928 06:45:32.132584 12767 raft_consensus.cc:388] T 00000000000000000000000000000000 P 543cea5c4e5849ab876e36f510ccb56a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:45:32.132648 12767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 543cea5c4e5849ab876e36f510ccb56a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:45:32.132771 12767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 543cea5c4e5849ab876e36f510ccb56a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:32.133642 12767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 543cea5c4e5849ab876e36f510ccb56a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543cea5c4e5849ab876e36f510ccb56a" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 38141 } }
I20260928 06:45:32.133978 12767 leader_election.cc:302] T 00000000000000000000000000000000 P 543cea5c4e5849ab876e36f510ccb56a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 543cea5c4e5849ab876e36f510ccb56a; no voters:
I20260928 06:45:32.134294 12767 leader_election.cc:288] T 00000000000000000000000000000000 P 543cea5c4e5849ab876e36f510ccb56a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:45:32.134393 12772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 543cea5c4e5849ab876e36f510ccb56a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:32.134665 12772 raft_consensus.cc:686] T 00000000000000000000000000000000 P 543cea5c4e5849ab876e36f510ccb56a [term 1 LEADER]: Becoming Leader. State: Replica: 543cea5c4e5849ab876e36f510ccb56a, State: Running, Role: LEADER
I20260928 06:45:32.134999 12772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 543cea5c4e5849ab876e36f510ccb56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543cea5c4e5849ab876e36f510ccb56a" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 38141 } }
I20260928 06:45:32.135563 12767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 543cea5c4e5849ab876e36f510ccb56a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:45:32.136696 12773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 543cea5c4e5849ab876e36f510ccb56a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "543cea5c4e5849ab876e36f510ccb56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543cea5c4e5849ab876e36f510ccb56a" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 38141 } } }
I20260928 06:45:32.136696 12774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 543cea5c4e5849ab876e36f510ccb56a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 543cea5c4e5849ab876e36f510ccb56a. Latest consensus state: current_term: 1 leader_uuid: "543cea5c4e5849ab876e36f510ccb56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543cea5c4e5849ab876e36f510ccb56a" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 38141 } } }
I20260928 06:45:32.136850 12774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 543cea5c4e5849ab876e36f510ccb56a [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:32.136850 12773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 543cea5c4e5849ab876e36f510ccb56a [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:32.137298 12781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:45:32.140377 12781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:45:32.146664 12781 catalog_manager.cc:1347] Generated new cluster ID: 1adc619a8c3749278b6442a193c01b93
I20260928 06:45:32.146747 12781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:45:32.180042 12781 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:45:32.181061 12781 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:45:32.185747 12781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 543cea5c4e5849ab876e36f510ccb56a: Generated new TSK 0
I20260928 06:45:32.186643 12781 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:45:32.210091 12770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:32.204834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52166 (local address 127.7.48.62:38141)
0928 06:45:32.204978 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:32.204984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:32.205165 (+ 181us) server_negotiation.cc:407] Connection header received
0928 06:45:32.205328 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:32.205330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:32.205378 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:32.205469 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:45:32.205738 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:32.206264 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:32.207481 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:32.207695 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:32.207768 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:32.208190 (+ 422us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:45:32.209680 (+ 1490us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33411/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 06:45:32.209937 (+ 257us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52166: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33411/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 06:45:32.210167 12770 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52166: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33411/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 06:45:32.212033 12796 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:32.204742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:38141 (local address 127.0.0.1:52166)
0928 06:45:32.205073 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:32.205089 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:32.205203 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:32.205503 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:32.205506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:32.205515 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:45:32.205631 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:32.205638 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:32.206389 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:32.206393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:32.207366 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:32.207376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:32.207794 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:32.207818 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:32.207929 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:32.207964 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:45:32.209866 (+ 1902us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33411/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 06:45:32.211813 (+ 1947us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.48.62:38141: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33411/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":1736,"thread_start_us":106,"threads_started":1}
I20260928 06:45:32.212898 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 12700
E20260928 06:45:32.223246 12797 webserver.cc:533] Webserver: sslize failed:
E20260928 06:45:32.223318 12797 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T06:45:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (325 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T06:45:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:45:32Z Disabled control of system clock
I20260928 06:45:32.263284 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:38291
--webserver_interface=127.7.48.62
--webserver_port=0
--builtin_ntp_servers=127.7.48.20:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:38291
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577905526996-7360-0/testchainca.cert with env {}
W20260928 06:45:32.374455 12801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:32.374773 12801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:32.374876 12801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:32.378960 12801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:45:32.379051 12801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:32.379081 12801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:32.379123 12801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:45:32.379168 12801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:45:32.379206 12801 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:45:32.379238 12801 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 06:45:32.384605 12801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:38291
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:38291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577905526996-7360-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:32.385692 12801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:32.386727 12801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:32.395417 12809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:32.395473 12801 server_base.cc:1034] running on GCE node
W20260928 06:45:32.395421 12807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:32.395417 12806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:32.396301 12801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:32.397413 12801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:32.398587 12801 hybrid_clock.cc:648] HybridClock initialized: now 1790577932398574 us; error 27 us; skew 500 ppm
I20260928 06:45:32.400499 12801 webserver.cc:460] Webserver started at http://127.7.48.62:41827/ using document root <none> and password file <none>
I20260928 06:45:32.401104 12801 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:32.401167 12801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:32.401458 12801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:32.403556 12801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "4b738ad9e34e464f8c3bd6296e3e1e1b"
format_stamp: "Formatted at 2026-09-28 06:45:32 on dist-test-slave-twl6"
server_key: "ba575a3438fabb9e013433157dd493d3"
server_key_iv: "c736bd57ad2f950436ce269b8de54a1d"
server_key_version: "encryptionkey@0"
I20260928 06:45:32.404146 12801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "4b738ad9e34e464f8c3bd6296e3e1e1b"
format_stamp: "Formatted at 2026-09-28 06:45:32 on dist-test-slave-twl6"
server_key: "ba575a3438fabb9e013433157dd493d3"
server_key_iv: "c736bd57ad2f950436ce269b8de54a1d"
server_key_version: "encryptionkey@0"
I20260928 06:45:32.407869 12801 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 06:45:32.410501 12815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:32.411768 12801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:32.411949 12801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "4b738ad9e34e464f8c3bd6296e3e1e1b"
format_stamp: "Formatted at 2026-09-28 06:45:32 on dist-test-slave-twl6"
server_key: "ba575a3438fabb9e013433157dd493d3"
server_key_iv: "c736bd57ad2f950436ce269b8de54a1d"
server_key_version: "encryptionkey@0"
I20260928 06:45:32.412081 12801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:32.445777 12801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:32.446578 12801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:32.446800 12801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:32.454304 12867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:38291 every 8 connection(s)
I20260928 06:45:32.454320 12801 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:38291
I20260928 06:45:32.455462 12801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:45:32.458390 12868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:32.458928 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 12801
I20260928 06:45:32.459077 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:45:32.459374 7360 external_mini_cluster.cc:1428] Setting key 907d701e12d091b42b1e193f57feb9f9
I20260928 06:45:32.464716 12868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b738ad9e34e464f8c3bd6296e3e1e1b: Bootstrap starting.
I20260928 06:45:32.467099 12868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b738ad9e34e464f8c3bd6296e3e1e1b: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:32.467954 12868 log.cc:824] T 00000000000000000000000000000000 P 4b738ad9e34e464f8c3bd6296e3e1e1b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:32.469115 12871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:32.460819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57770 (local address 127.7.48.62:38291)
0928 06:45:32.461473 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:32.461489 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:32.461531 (+ 42us) server_negotiation.cc:407] Connection header received
0928 06:45:32.462179 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:32.462198 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:32.462605 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:32.462951 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:32.463391 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:32.464433 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:32.465389 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:32.465855 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:32.466035 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:32.466938 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:32.466943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:32.466955 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:45:32.467397 (+ 442us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:32.467456 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:45:32.467477 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:32.467592 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:32.467785 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 06:45:32.468107 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":145,"threads_started":1}
I20260928 06:45:32.470093 12868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b738ad9e34e464f8c3bd6296e3e1e1b: No bootstrap required, opened a new log
I20260928 06:45:32.471972 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:45:32.474159 12868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b738ad9e34e464f8c3bd6296e3e1e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b738ad9e34e464f8c3bd6296e3e1e1b" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 38291 } }
I20260928 06:45:32.474403 12868 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b738ad9e34e464f8c3bd6296e3e1e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:32.474494 12868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b738ad9e34e464f8c3bd6296e3e1e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b738ad9e34e464f8c3bd6296e3e1e1b, State: Initialized, Role: FOLLOWER
I20260928 06:45:32.474982 12868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b738ad9e34e464f8c3bd6296e3e1e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b738ad9e34e464f8c3bd6296e3e1e1b" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 38291 } }
I20260928 06:45:32.475135 12868 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b738ad9e34e464f8c3bd6296e3e1e1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:45:32.475204 12868 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b738ad9e34e464f8c3bd6296e3e1e1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:45:32.475328 12868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b738ad9e34e464f8c3bd6296e3e1e1b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:32.476333 12868 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b738ad9e34e464f8c3bd6296e3e1e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b738ad9e34e464f8c3bd6296e3e1e1b" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 38291 } }
I20260928 06:45:32.476691 12868 leader_election.cc:302] T 00000000000000000000000000000000 P 4b738ad9e34e464f8c3bd6296e3e1e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b738ad9e34e464f8c3bd6296e3e1e1b; no voters:
I20260928 06:45:32.477001 12868 leader_election.cc:288] T 00000000000000000000000000000000 P 4b738ad9e34e464f8c3bd6296e3e1e1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:45:32.477100 12873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b738ad9e34e464f8c3bd6296e3e1e1b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:32.477406 12873 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b738ad9e34e464f8c3bd6296e3e1e1b [term 1 LEADER]: Becoming Leader. State: Replica: 4b738ad9e34e464f8c3bd6296e3e1e1b, State: Running, Role: LEADER
I20260928 06:45:32.477771 12873 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b738ad9e34e464f8c3bd6296e3e1e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b738ad9e34e464f8c3bd6296e3e1e1b" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 38291 } }
I20260928 06:45:32.478267 12868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b738ad9e34e464f8c3bd6296e3e1e1b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:45:32.479480 12875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b738ad9e34e464f8c3bd6296e3e1e1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b738ad9e34e464f8c3bd6296e3e1e1b. Latest consensus state: current_term: 1 leader_uuid: "4b738ad9e34e464f8c3bd6296e3e1e1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b738ad9e34e464f8c3bd6296e3e1e1b" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 38291 } } }
I20260928 06:45:32.479614 12875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b738ad9e34e464f8c3bd6296e3e1e1b [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:32.480018 12882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:45:32.480194 12874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b738ad9e34e464f8c3bd6296e3e1e1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b738ad9e34e464f8c3bd6296e3e1e1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b738ad9e34e464f8c3bd6296e3e1e1b" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 38291 } } }
I20260928 06:45:32.480342 12874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b738ad9e34e464f8c3bd6296e3e1e1b [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:32.483218 12882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:45:32.489197 12882 catalog_manager.cc:1347] Generated new cluster ID: 21add71b362c459ba44c76246307fa26
I20260928 06:45:32.489274 12882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:45:32.502359 12882 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:45:32.503338 12882 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:45:32.513239 12882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b738ad9e34e464f8c3bd6296e3e1e1b: Generated new TSK 0
I20260928 06:45:32.513967 12882 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:45:32.586802 12871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:32.532758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57784 (local address 127.7.48.62:38291)
0928 06:45:32.532915 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:32.532921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:32.533413 (+ 492us) server_negotiation.cc:407] Connection header received
0928 06:45:32.533626 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:32.533631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:32.533697 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:32.533828 (+ 131us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:45:32.534122 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:32.535113 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:32.536622 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:32.536975 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:32.537042 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:32.537548 (+ 506us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:45:32.586292 (+ 48744us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 06:45:32.586629 (+ 337us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57784: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 06:45:32.586901 12871 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57784: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 06:45:32.586946 12897 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:32.532640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:38291 (local address 127.0.0.1:57784)
0928 06:45:32.533304 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:32.533322 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:32.533473 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:32.533850 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:32.533853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:32.533861 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:45:32.533991 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:32.534014 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:32.535246 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:32.535249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:32.536463 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:32.536478 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:32.537163 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:32.537192 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:32.537306 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:32.537340 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:45:32.586548 (+ 49208us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 06:45:32.586747 (+ 199us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.48.62:38291: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":99,"threads_started":1}
I20260928 06:45:32.587611 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 12801
2026-09-28T06:45:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (351 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (126 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (124 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:45:32 dist-test-slave-twl6 krb5kdc[12909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:45:32 dist-test-slave-twl6 krb5kdc[12909](info): set up 2 sockets
Sep 28 06:45:32 dist-test-slave-twl6 krb5kdc[12909](info): commencing operation
krb5kdc: starting...
W20260928 06:45:33.868620 7360 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:45:33 dist-test-slave-twl6 krb5kdc[12909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:45:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:45:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:45:33.990895 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:43871
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:43921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:43871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:34.106706 12922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:34.107049 12922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:34.107143 12922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:34.111217 12922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:45:34.111325 12922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:34.111364 12922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:34.111424 12922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:45:34.111464 12922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:45:34.116932 12922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:43921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:43871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:43871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:34.118141 12922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:34.119212 12922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:34.127848 12927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:34.127830 12930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:34.128033 12922 server_base.cc:1034] running on GCE node
W20260928 06:45:34.127830 12928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:34.128711 12922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:34.129882 12922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:34.131047 12922 hybrid_clock.cc:648] HybridClock initialized: now 1790577934131013 us; error 47 us; skew 500 ppm
Sep 28 06:45:34 dist-test-slave-twl6 krb5kdc[12909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577934, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:34.140908 12922 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:45:34.141954 12922 webserver.cc:460] Webserver started at http://127.7.48.62:41537/ using document root <none> and password file <none>
I20260928 06:45:34.142606 12922 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:34.142709 12922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:34.142971 12922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:34.145242 12922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "63367035fbea4aa4a90647da53237cda"
format_stamp: "Formatted at 2026-09-28 06:45:34 on dist-test-slave-twl6"
server_key: "3fb24300e22ec95e536823cd3b9abfcd"
server_key_iv: "2d660fa89a072b8ffed1305143b3a2da"
server_key_version: "encryptionkey@0"
I20260928 06:45:34.145834 12922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "63367035fbea4aa4a90647da53237cda"
format_stamp: "Formatted at 2026-09-28 06:45:34 on dist-test-slave-twl6"
server_key: "3fb24300e22ec95e536823cd3b9abfcd"
server_key_iv: "2d660fa89a072b8ffed1305143b3a2da"
server_key_version: "encryptionkey@0"
I20260928 06:45:34.149989 12922 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260928 06:45:34.152563 12937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:34.153839 12922 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:45:34.153996 12922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "63367035fbea4aa4a90647da53237cda"
format_stamp: "Formatted at 2026-09-28 06:45:34 on dist-test-slave-twl6"
server_key: "3fb24300e22ec95e536823cd3b9abfcd"
server_key_iv: "2d660fa89a072b8ffed1305143b3a2da"
server_key_version: "encryptionkey@0"
I20260928 06:45:34.154114 12922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:34.172230 12922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:34.175163 12922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:34.175376 12922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:34.182852 12922 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:43871
I20260928 06:45:34.182880 12989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:43871 every 8 connection(s)
I20260928 06:45:34.183838 12922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:45:34.186693 12990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:34.187078 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 12922
I20260928 06:45:34.187214 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:45:34.187505 7360 external_mini_cluster.cc:1428] Setting key 1598692ac804e374794209e711b095e7
I20260928 06:45:34.194156 12990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63367035fbea4aa4a90647da53237cda: Bootstrap starting.
I20260928 06:45:34.196954 12990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63367035fbea4aa4a90647da53237cda: Neither blocks nor log segments found. Creating new log.
Sep 28 06:45:34 dist-test-slave-twl6 krb5kdc[12909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:34.197903 12990 log.cc:824] T 00000000000000000000000000000000 P 63367035fbea4aa4a90647da53237cda: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:34.200255 12990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63367035fbea4aa4a90647da53237cda: No bootstrap required, opened a new log
I20260928 06:45:34.202754 12993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:34.189096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54334 (local address 127.7.48.62:43871)
0928 06:45:34.189539 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:34.189558 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:34.189605 (+ 47us) server_negotiation.cc:407] Connection header received
0928 06:45:34.190335 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:34.190361 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:34.190805 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:34.191185 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:34.191933 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:34.193471 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:34.194381 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:34.195052 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:34.195207 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:34.198118 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:34.198123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:34.198143 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:34.198171 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:34.199849 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:34.200446 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:34.200456 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:34.200461 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:34.200566 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:34.200864 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:34.200866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:34.200867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:34.201238 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:34.201375 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:34.201687 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 06:45:34.201994 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":123,"threads_started":1}
I20260928 06:45:34.204342 12990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63367035fbea4aa4a90647da53237cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63367035fbea4aa4a90647da53237cda" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 43871 } }
I20260928 06:45:34.204625 12990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63367035fbea4aa4a90647da53237cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:34.204687 12990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63367035fbea4aa4a90647da53237cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63367035fbea4aa4a90647da53237cda, State: Initialized, Role: FOLLOWER
I20260928 06:45:34.205176 12990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63367035fbea4aa4a90647da53237cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63367035fbea4aa4a90647da53237cda" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 43871 } }
I20260928 06:45:34.205348 12990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63367035fbea4aa4a90647da53237cda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:45:34.205399 12990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63367035fbea4aa4a90647da53237cda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:45:34.205472 12990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63367035fbea4aa4a90647da53237cda [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:34.206326 12990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63367035fbea4aa4a90647da53237cda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63367035fbea4aa4a90647da53237cda" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 43871 } }
I20260928 06:45:34.206631 12990 leader_election.cc:302] T 00000000000000000000000000000000 P 63367035fbea4aa4a90647da53237cda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63367035fbea4aa4a90647da53237cda; no voters:
I20260928 06:45:34.206903 12990 leader_election.cc:288] T 00000000000000000000000000000000 P 63367035fbea4aa4a90647da53237cda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:45:34.207038 12995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63367035fbea4aa4a90647da53237cda [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:34.207289 12995 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63367035fbea4aa4a90647da53237cda [term 1 LEADER]: Becoming Leader. State: Replica: 63367035fbea4aa4a90647da53237cda, State: Running, Role: LEADER
I20260928 06:45:34.207615 12995 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63367035fbea4aa4a90647da53237cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63367035fbea4aa4a90647da53237cda" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 43871 } }
I20260928 06:45:34.208199 12990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63367035fbea4aa4a90647da53237cda [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:45:34.209393 12996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63367035fbea4aa4a90647da53237cda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63367035fbea4aa4a90647da53237cda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63367035fbea4aa4a90647da53237cda" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 43871 } } }
I20260928 06:45:34.209393 12997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63367035fbea4aa4a90647da53237cda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63367035fbea4aa4a90647da53237cda. Latest consensus state: current_term: 1 leader_uuid: "63367035fbea4aa4a90647da53237cda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63367035fbea4aa4a90647da53237cda" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 43871 } } }
I20260928 06:45:34.209599 12997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63367035fbea4aa4a90647da53237cda [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:34.209599 12996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63367035fbea4aa4a90647da53237cda [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:34.210031 13004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:45:34.212949 13004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:45:34.218910 13004 catalog_manager.cc:1347] Generated new cluster ID: 2a80a8d1d1c14b43b6fda85ae719a99a
I20260928 06:45:34.218983 13004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:45:34.231869 13004 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:45:34.232921 13004 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:45:34.238152 13004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63367035fbea4aa4a90647da53237cda: Generated new TSK 0
I20260928 06:45:34.238878 13004 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
I20260928 06:45:34.275594 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:43871
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:43921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:34.389914 13016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:34.390271 13016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:34.390348 13016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:34.394495 13016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:34.394588 13016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:34.394683 13016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:45:34.400030 13016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:43921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:43871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:34.401149 13016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:34.402184 13016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:34.412189 13021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:34.412181 13022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:34.412176 13024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:34.412389 13016 server_base.cc:1034] running on GCE node
I20260928 06:45:34.412842 13016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:34.413609 13016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:34.414788 13016 hybrid_clock.cc:648] HybridClock initialized: now 1790577934414769 us; error 34 us; skew 500 ppm
Sep 28 06:45:34 dist-test-slave-twl6 krb5kdc[12909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577934, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:34.417655 13016 init.cc:377] Logged in from keytab as kudu/127.7.48.1@KRBTEST.COM (short username kudu)
I20260928 06:45:34.418546 13016 webserver.cc:460] Webserver started at http://127.7.48.1:37273/ using document root <none> and password file <none>
I20260928 06:45:34.419082 13016 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:34.419142 13016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:34.419307 13016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:34.421419 13016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "5ae5b502d62b4149bc7bc94ceb79a521"
format_stamp: "Formatted at 2026-09-28 06:45:34 on dist-test-slave-twl6"
server_key: "5fc93e00d9bdc284d9370cce491a15e3"
server_key_iv: "639672329d37701cac57aa7ebc632335"
server_key_version: "encryptionkey@0"
I20260928 06:45:34.421957 13016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "5ae5b502d62b4149bc7bc94ceb79a521"
format_stamp: "Formatted at 2026-09-28 06:45:34 on dist-test-slave-twl6"
server_key: "5fc93e00d9bdc284d9370cce491a15e3"
server_key_iv: "639672329d37701cac57aa7ebc632335"
server_key_version: "encryptionkey@0"
I20260928 06:45:34.425760 13016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:45:34.428282 13031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:34.429528 13016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:45:34.429668 13016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "5ae5b502d62b4149bc7bc94ceb79a521"
format_stamp: "Formatted at 2026-09-28 06:45:34 on dist-test-slave-twl6"
server_key: "5fc93e00d9bdc284d9370cce491a15e3"
server_key_iv: "639672329d37701cac57aa7ebc632335"
server_key_version: "encryptionkey@0"
I20260928 06:45:34.429786 13016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:34.447626 13016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:34.451216 13016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:34.451431 13016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:34.452105 13016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:34.452937 13016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:34.452984 13016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:34.453024 13016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:34.453042 13016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:34.462633 13016 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:36677
I20260928 06:45:34.462663 13144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:36677 every 8 connection(s)
I20260928 06:45:34.463575 13016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:45:34.472680 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 13016
I20260928 06:45:34.472815 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
I20260928 06:45:34.473110 7360 external_mini_cluster.cc:1428] Setting key 75e3142af397e8aef31d26e463303fc9
Sep 28 06:45:34 dist-test-slave-twl6 krb5kdc[12909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577934, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:34.477726 12993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:34.465498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:43545 (local address 127.7.48.62:43871)
0928 06:45:34.465648 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:34.465653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:34.466539 (+ 886us) server_negotiation.cc:407] Connection header received
0928 06:45:34.467849 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:34.467852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:34.467961 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:34.468036 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:34.469224 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:34.469759 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:34.470846 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:34.471060 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:34.471135 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:34.473852 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:34.473856 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:34.473861 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:34.473893 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:34.475546 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:34.476142 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:34.476149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:34.476154 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:34.476251 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:34.476601 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:34.476605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:34.476607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:34.476798 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:34.476908 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:34.477427 (+ 519us) server_negotiation.cc:299] Negotiation successful
0928 06:45:34.477563 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 06:45:34.478596 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:34.465930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:43871 (local address 127.7.48.1:43545)
0928 06:45:34.466389 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:34.466434 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:34.467659 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:34.468175 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:34.468183 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:34.468547 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:34.469056 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:34.469077 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:34.469884 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:34.469886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:34.470744 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:34.470752 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:34.471188 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:34.471191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:34.471327 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:34.471999 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:34.472016 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:34.473682 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:34.475674 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:34.475678 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:34.475692 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:34.476036 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:34.476341 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:34.476344 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:34.476345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:34.476480 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:34.476908 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:34.476917 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:34.477189 (+ 272us) client_negotiation.cc:769] Sending connection context
0928 06:45:34.477387 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 06:45:34.477742 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":121,"threads_started":1}
I20260928 06:45:34.479907 13145 heartbeater.cc:347] Connected to a master server at 127.7.48.62:43871
I20260928 06:45:34.480203 13145 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:34.480873 13145 heartbeater.cc:511] Master 127.7.48.62:43871 requested a full tablet report, sending...
I20260928 06:45:34.482738 12954 ts_manager.cc:194] Registered new tserver with Master: 5ae5b502d62b4149bc7bc94ceb79a521 (127.7.48.1:36677)
WARNING: no policy specified for kudu/127.7.48.2@KRBTEST.COM; defaulting to no policy
I20260928 06:45:34.484246 12954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.1@KRBTEST.COM'} at 127.7.48.1:43545
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
I20260928 06:45:34.503311 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:43871
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:43921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:34.616977 13150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:34.617249 13150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:34.617292 13150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:34.621163 13150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:34.621241 13150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:34.621299 13150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:45:34.626724 13150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:43921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:43871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:34.627743 13150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:34.628818 13150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:34.638552 13158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:34.638617 13150 server_base.cc:1034] running on GCE node
W20260928 06:45:34.638520 13155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:34.638547 13156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:34.639348 13150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:34.640023 13150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:34.641181 13150 hybrid_clock.cc:648] HybridClock initialized: now 1790577934641166 us; error 27 us; skew 500 ppm
Sep 28 06:45:34 dist-test-slave-twl6 krb5kdc[12909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577934, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:34.643998 13150 init.cc:377] Logged in from keytab as kudu/127.7.48.2@KRBTEST.COM (short username kudu)
I20260928 06:45:34.644912 13150 webserver.cc:460] Webserver started at http://127.7.48.2:41329/ using document root <none> and password file <none>
I20260928 06:45:34.645490 13150 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:34.645577 13150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:34.645802 13150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:34.647820 13150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "16d4225ce53f4eb1963088236cb16792"
format_stamp: "Formatted at 2026-09-28 06:45:34 on dist-test-slave-twl6"
server_key: "b9a5eebcd3fee5722834df1b5833a4f6"
server_key_iv: "07977281ff464cd637c87cce782d7caf"
server_key_version: "encryptionkey@0"
I20260928 06:45:34.648435 13150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "16d4225ce53f4eb1963088236cb16792"
format_stamp: "Formatted at 2026-09-28 06:45:34 on dist-test-slave-twl6"
server_key: "b9a5eebcd3fee5722834df1b5833a4f6"
server_key_iv: "07977281ff464cd637c87cce782d7caf"
server_key_version: "encryptionkey@0"
I20260928 06:45:34.652221 13150 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 06:45:34.654657 13165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:34.655865 13150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:34.656035 13150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "16d4225ce53f4eb1963088236cb16792"
format_stamp: "Formatted at 2026-09-28 06:45:34 on dist-test-slave-twl6"
server_key: "b9a5eebcd3fee5722834df1b5833a4f6"
server_key_iv: "07977281ff464cd637c87cce782d7caf"
server_key_version: "encryptionkey@0"
I20260928 06:45:34.656157 13150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:34.679620 13150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:34.682760 13150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:34.682956 13150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:34.683568 13150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:34.684417 13150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:34.684484 13150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:34.684553 13150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:34.684593 13150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:34.693892 13150 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:41121
I20260928 06:45:34.693919 13278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:41121 every 8 connection(s)
I20260928 06:45:34.694802 13150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:45:34.699961 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 13150
I20260928 06:45:34.700088 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:45:34.700357 7360 external_mini_cluster.cc:1428] Setting key 938fc496f9d4cf58021ef53172198edc
Sep 28 06:45:34 dist-test-slave-twl6 krb5kdc[12909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577934, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.48.3@KRBTEST.COM; defaulting to no policy
I20260928 06:45:34.710291 12993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:34.696722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:60561 (local address 127.7.48.62:43871)
0928 06:45:34.696870 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:34.696875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:34.697792 (+ 917us) server_negotiation.cc:407] Connection header received
0928 06:45:34.699211 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:34.699214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:34.699258 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:34.699332 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:34.700720 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:34.701360 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:34.702872 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:34.703289 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:34.703421 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:34.706227 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:34.706230 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:34.706232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:34.706255 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:34.707723 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:34.708402 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:34.708408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:34.708410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:34.708516 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:34.708912 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:34.708915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:34.708917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:34.709186 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:34.709302 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:34.709941 (+ 639us) server_negotiation.cc:299] Negotiation successful
0928 06:45:34.710081 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 06:45:34.711233 13281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:34.697174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:43871 (local address 127.7.48.2:60561)
0928 06:45:34.697635 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:34.697682 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:34.698999 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:34.699495 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:34.699501 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:34.699955 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:34.700570 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:34.700586 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:34.701499 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:34.701503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:34.702711 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:34.702724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:34.703474 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:34.703478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:34.703656 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:34.704322 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:34.704338 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:34.706044 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:34.707853 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:34.707859 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:34.707903 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:34.708233 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:34.708614 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:34.708617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:34.708618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:34.708755 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:34.709319 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:34.709326 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:34.709629 (+ 303us) client_negotiation.cc:769] Sending connection context
0928 06:45:34.709839 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 06:45:34.710310 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":113,"threads_started":1}
I20260928 06:45:34.712430 13279 heartbeater.cc:347] Connected to a master server at 127.7.48.62:43871
I20260928 06:45:34.712677 13279 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.3@KRBTEST.COM" created.
I20260928 06:45:34.713263 13279 heartbeater.cc:511] Master 127.7.48.62:43871 requested a full tablet report, sending...
I20260928 06:45:34.714471 12954 ts_manager.cc:194] Registered new tserver with Master: 16d4225ce53f4eb1963088236cb16792 (127.7.48.2:41121)
I20260928 06:45:34.715078 12954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.2@KRBTEST.COM'} at 127.7.48.2:60561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
I20260928 06:45:34.728348 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:43871
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:43921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:34.841161 13284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:34.841410 13284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:34.841451 13284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:34.845498 13284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:34.845559 13284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:34.845654 13284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:45:34.851017 13284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:43921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:43871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:34.852155 13284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:34.853247 13284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:34.863058 13289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:34.863106 13290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:34.863058 13292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:34.863309 13284 server_base.cc:1034] running on GCE node
I20260928 06:45:34.863920 13284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:34.864573 13284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:34.865760 13284 hybrid_clock.cc:648] HybridClock initialized: now 1790577934865731 us; error 48 us; skew 500 ppm
Sep 28 06:45:34 dist-test-slave-twl6 krb5kdc[12909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577934, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:34.868597 13284 init.cc:377] Logged in from keytab as kudu/127.7.48.3@KRBTEST.COM (short username kudu)
I20260928 06:45:34.869397 13284 webserver.cc:460] Webserver started at http://127.7.48.3:46601/ using document root <none> and password file <none>
I20260928 06:45:34.869913 13284 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:34.869969 13284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:34.870129 13284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:34.872144 13284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "8d22474ab4d04e0d90a1e5db9fd15ef4"
format_stamp: "Formatted at 2026-09-28 06:45:34 on dist-test-slave-twl6"
server_key: "cd767e1426158e94cfcdbb502059d28a"
server_key_iv: "abebbf22890fe75abfa717ff6fd19c30"
server_key_version: "encryptionkey@0"
I20260928 06:45:34.872658 13284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "8d22474ab4d04e0d90a1e5db9fd15ef4"
format_stamp: "Formatted at 2026-09-28 06:45:34 on dist-test-slave-twl6"
server_key: "cd767e1426158e94cfcdbb502059d28a"
server_key_iv: "abebbf22890fe75abfa717ff6fd19c30"
server_key_version: "encryptionkey@0"
I20260928 06:45:34.876246 13284 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:45:34.878494 13299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:34.879702 13284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:34.879798 13284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "8d22474ab4d04e0d90a1e5db9fd15ef4"
format_stamp: "Formatted at 2026-09-28 06:45:34 on dist-test-slave-twl6"
server_key: "cd767e1426158e94cfcdbb502059d28a"
server_key_iv: "abebbf22890fe75abfa717ff6fd19c30"
server_key_version: "encryptionkey@0"
I20260928 06:45:34.879917 13284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:34.900111 13284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:34.903430 13284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:34.903597 13284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:34.904166 13284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:34.904985 13284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:34.905033 13284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:34.905073 13284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:34.905093 13284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:34.914134 13284 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:43867
I20260928 06:45:34.914201 13412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:43867 every 8 connection(s)
I20260928 06:45:34.915046 13284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:45:34 dist-test-slave-twl6 krb5kdc[12909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577934, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:34.924537 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 13284
I20260928 06:45:34.924676 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:45:34.924990 7360 external_mini_cluster.cc:1428] Setting key e75c543e0c3fa4bee5e7917a0a73f8a0
I20260928 06:45:34.929051 12993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:34.916993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:36239 (local address 127.7.48.62:43871)
0928 06:45:34.917139 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:34.917143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:34.917946 (+ 803us) server_negotiation.cc:407] Connection header received
0928 06:45:34.919203 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:34.919205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:34.919251 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:34.919325 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:34.920656 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:34.921217 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:34.922261 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:34.922463 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:34.922543 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:34.925058 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:34.925063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:34.925066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:34.925095 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:34.926785 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:34.927393 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:34.927399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:34.927402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:34.927519 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:34.928046 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:34.928052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:34.928056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:34.928305 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:34.928388 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:34.928840 (+ 452us) server_negotiation.cc:299] Negotiation successful
0928 06:45:34.928937 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:45:34.929912 13415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:34.917356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:43871 (local address 127.7.48.3:36239)
0928 06:45:34.917794 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:34.917834 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:34.919006 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:34.919482 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:34.919488 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:34.919865 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:34.920506 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:34.920521 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:34.921320 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:34.921322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:34.922161 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:34.922169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:34.922589 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:34.922591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:34.922725 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:34.923243 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:34.923259 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:34.924857 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:34.926897 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:34.926903 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:34.926917 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:34.927282 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:34.927651 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:34.927654 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:34.927656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:34.927922 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:34.928403 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:34.928407 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:34.928634 (+ 227us) client_negotiation.cc:769] Sending connection context
0928 06:45:34.928780 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 06:45:34.929020 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":120,"threads_started":1}
I20260928 06:45:34.930985 13413 heartbeater.cc:347] Connected to a master server at 127.7.48.62:43871
I20260928 06:45:34.931232 13413 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:34.931713 13413 heartbeater.cc:511] Master 127.7.48.62:43871 requested a full tablet report, sending...
I20260928 06:45:34.932694 12954 ts_manager.cc:194] Registered new tserver with Master: 8d22474ab4d04e0d90a1e5db9fd15ef4 (127.7.48.3:43867)
I20260928 06:45:34.933245 12954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.3@KRBTEST.COM'} at 127.7.48.3:36239
I20260928 06:45:34.938725 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/test-admin.keytab.
Sep 28 06:45:34 dist-test-slave-twl6 krb5kdc[12909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:34.955844 7360 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 06:45:34.959004 13420 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:34.957385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58938 (local address 127.7.48.1:36677)
0928 06:45:34.957781 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:34.957796 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:34.957821 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:45:34.957900 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:34.957907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:34.958115 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:34.958265 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:34.958810 (+ 545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58938: BlockingRecv error: recv got EOF from 127.0.0.1:58938 (error 108)
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":87,"threads_started":1}
W20260928 06:45:34.959088 7360 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/krb5cc' of type FILE
Sep 28 06:45:34 dist-test-slave-twl6 krb5kdc[12909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:34.960246 7360 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 06:45:34 dist-test-slave-twl6 krb5kdc[12909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:34.961138 7360 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 06:45:34 dist-test-slave-twl6 krb5kdc[12909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.1@KRBTEST.COM
I20260928 06:45:34.970831 13420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:34.962034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58942 (local address 127.7.48.1:36677)
0928 06:45:34.962195 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:34.962200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:34.962292 (+ 92us) server_negotiation.cc:407] Connection header received
0928 06:45:34.962433 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:34.962436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:34.962484 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:34.962566 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:34.963083 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:34.963868 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:34.964795 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:34.965203 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:34.965314 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:34.967465 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:34.967474 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:34.967487 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:34.967510 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:34.969007 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:34.969471 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:34.969477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:34.969496 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:34.969570 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:34.969862 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:34.969865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:34.969867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:34.970106 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:34.970327 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:34.970512 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 06:45:34.970683 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 06:45:34.974754 13420 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:34.973742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58950 (local address 127.7.48.1:36677)
0928 06:45:34.973869 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:34.973874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:34.973963 (+ 89us) server_negotiation.cc:407] Connection header received
0928 06:45:34.974106 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:34.974109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:34.974160 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:34.974239 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:34.974640 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58950: BlockingRecv error: recv got EOF from 127.0.0.1:58950 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 06:45:34.974893 7360 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/krb5cc' of type FILE
Sep 28 06:45:34 dist-test-slave-twl6 krb5kdc[12909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:34.975981 7360 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 06:45:34 dist-test-slave-twl6 krb5kdc[12909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:34.976835 7360 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 06:45:34 dist-test-slave-twl6 krb5kdc[12909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.1@KRBTEST.COM
I20260928 06:45:34.985456 13420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:34.977567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58954 (local address 127.7.48.1:36677)
0928 06:45:34.977684 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:34.977688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:34.977776 (+ 88us) server_negotiation.cc:407] Connection header received
0928 06:45:34.977913 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:34.977916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:34.977959 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:34.978041 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:34.978527 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:34.979014 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:34.979928 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:34.980142 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:34.980246 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:34.982315 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:34.982321 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:34.982328 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:34.982350 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:34.983805 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:34.984280 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:34.984284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:34.984285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:34.984366 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:34.984673 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:34.984677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:34.984678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:34.984838 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:34.984968 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:34.985203 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 06:45:34.985305 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 06:45:34.988113 13420 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:34.986983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58962 (local address 127.7.48.1:36677)
0928 06:45:34.987108 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:34.987111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:34.987192 (+ 81us) server_negotiation.cc:407] Connection header received
0928 06:45:34.987300 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:34.987303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:34.987345 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:34.987456 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:34.987982 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58962: BlockingRecv error: recv got EOF from 127.0.0.1:58962 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 06:45:34.988247 7360 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577905526996-7360-0/krb5kdc/krb5cc' of type FILE
Sep 28 06:45:34 dist-test-slave-twl6 krb5kdc[12909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:34.989240 7360 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 06:45:34 dist-test-slave-twl6 krb5kdc[12909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:34.990082 7360 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 06:45:34 dist-test-slave-twl6 krb5kdc[12909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.1@KRBTEST.COM
I20260928 06:45:34.997956 13420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:34.990797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58964 (local address 127.7.48.1:36677)
0928 06:45:34.990908 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:34.990910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:34.990954 (+ 44us) server_negotiation.cc:407] Connection header received
0928 06:45:34.991041 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:34.991043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:34.991072 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:34.991133 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:34.991570 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:34.992177 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:34.992914 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:34.993110 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:34.993181 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:34.995272 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:34.995275 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:34.995277 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:34.995296 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:34.996535 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:34.996917 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:34.996920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:34.996921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:34.996988 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:34.997217 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:34.997219 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:34.997220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:34.997363 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:34.997453 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:34.997673 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 06:45:34.997783 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 06:45:34.998983 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 13016
I20260928 06:45:35.004845 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 13150
I20260928 06:45:35.010352 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 13284
I20260928 06:45:35.015614 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 12922
2026-09-28T06:45:35Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2179 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:45:35 dist-test-slave-twl6 krb5kdc[13433](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:45:35 dist-test-slave-twl6 krb5kdc[13433](info): set up 2 sockets
Sep 28 06:45:35 dist-test-slave-twl6 krb5kdc[13433](info): commencing operation
krb5kdc: starting...
W20260928 06:45:36.044894 7360 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:45:36 dist-test-slave-twl6 krb5kdc[13433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:45:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:45:36Z Disabled control of system clock
WARNING: no policy specified for oryx/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.62.keytab.
Entry for principal oryx/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.62.keytab.
I20260928 06:45:36.166365 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:34479
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.62.keytab
--principal=oryx/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:44951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:34479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:36.277701 13446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:36.278023 13446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:36.278107 13446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:36.282186 13446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:45:36.282276 13446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:36.282315 13446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:36.282354 13446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:45:36.282389 13446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:45:36.287760 13446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:44951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:34479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.62.keytab
--principal=oryx/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:34479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:36.288861 13446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:36.289906 13446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:36.298659 13452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:36.298653 13451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:36.298655 13454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:36.298995 13446 server_base.cc:1034] running on GCE node
I20260928 06:45:36.299595 13446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:36.300745 13446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:36.301944 13446 hybrid_clock.cc:648] HybridClock initialized: now 1790577936301932 us; error 32 us; skew 500 ppm
Sep 28 06:45:36 dist-test-slave-twl6 krb5kdc[13433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577936, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:36.304602 13446 init.cc:377] Logged in from keytab as oryx/127.7.48.62@KRBTEST.COM (short username oryx)
I20260928 06:45:36.305522 13446 webserver.cc:460] Webserver started at http://127.7.48.62:38709/ using document root <none> and password file <none>
I20260928 06:45:36.306089 13446 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:36.306180 13446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:36.306418 13446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:36.308554 13446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "1137a312ab134aa4afef56800ac54199"
format_stamp: "Formatted at 2026-09-28 06:45:36 on dist-test-slave-twl6"
server_key: "a3f922b6622f854614f221ea2e6e984d"
server_key_iv: "9d821a9841be0cb74321ad310ffa4c94"
server_key_version: "encryptionkey@0"
I20260928 06:45:36.309103 13446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "1137a312ab134aa4afef56800ac54199"
format_stamp: "Formatted at 2026-09-28 06:45:36 on dist-test-slave-twl6"
server_key: "a3f922b6622f854614f221ea2e6e984d"
server_key_iv: "9d821a9841be0cb74321ad310ffa4c94"
server_key_version: "encryptionkey@0"
I20260928 06:45:36.312799 13446 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 06:45:36.315201 13461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:36.316565 13446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:36.316725 13446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "1137a312ab134aa4afef56800ac54199"
format_stamp: "Formatted at 2026-09-28 06:45:36 on dist-test-slave-twl6"
server_key: "a3f922b6622f854614f221ea2e6e984d"
server_key_iv: "9d821a9841be0cb74321ad310ffa4c94"
server_key_version: "encryptionkey@0"
I20260928 06:45:36.316834 13446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:36.329672 13446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:36.332712 13446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:36.332906 13446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:36.339962 13446 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:34479
I20260928 06:45:36.340204 13513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:34479 every 8 connection(s)
I20260928 06:45:36.340948 13446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:45:36.343120 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 13446
I20260928 06:45:36.343240 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:45:36.343566 7360 external_mini_cluster.cc:1428] Setting key 89d3089c4805af6c3ed80bc00444b267
I20260928 06:45:36.344051 13514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:36.350106 13514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199: Bootstrap starting.
Sep 28 06:45:36 dist-test-slave-twl6 krb5kdc[13433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM
I20260928 06:45:36.352752 13514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:36.353552 13514 log.cc:824] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:36.355434 13514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199: No bootstrap required, opened a new log
I20260928 06:45:36.357841 13517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:36.345026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41916 (local address 127.7.48.62:34479)
0928 06:45:36.345482 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:36.345501 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:36.345538 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:45:36.346189 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:36.346207 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:36.346535 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:36.346869 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:36.347688 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:36.348772 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:36.349639 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:36.350111 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:36.350279 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:36.353126 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:36.353131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:36.353143 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:36.353170 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:36.354949 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:36.355404 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:36.355409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:36.355413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:36.355631 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:36.355962 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:36.355966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:36.355968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:36.356360 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:36.356584 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:36.356811 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 06:45:36.357120 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":142,"threads_started":1}
I20260928 06:45:36.359345 13514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } }
I20260928 06:45:36.359627 13514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:36.359692 13514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1137a312ab134aa4afef56800ac54199, State: Initialized, Role: FOLLOWER
I20260928 06:45:36.360180 13514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } }
I20260928 06:45:36.360350 13514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:45:36.360402 13514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:45:36.360476 13514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:36.361346 13514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } }
I20260928 06:45:36.361642 13514 leader_election.cc:302] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1137a312ab134aa4afef56800ac54199; no voters:
I20260928 06:45:36.361892 13514 leader_election.cc:288] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:45:36.362021 13519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:36.362265 13519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [term 1 LEADER]: Becoming Leader. State: Replica: 1137a312ab134aa4afef56800ac54199, State: Running, Role: LEADER
I20260928 06:45:36.362623 13519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } }
I20260928 06:45:36.363154 13514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:45:36.364174 13520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1137a312ab134aa4afef56800ac54199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } }
I20260928 06:45:36.364445 13521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1137a312ab134aa4afef56800ac54199. Latest consensus state: current_term: 1 leader_uuid: "1137a312ab134aa4afef56800ac54199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } }
I20260928 06:45:36.364574 13520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:36.364655 13521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:36.365674 13529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:45:36.368479 13529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:45:36.373767 13529 catalog_manager.cc:1347] Generated new cluster ID: 5736c5ca81164407b939a2e06709be0c
I20260928 06:45:36.373833 13529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:45:36.393029 13529 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:45:36.394059 13529 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:45:36.399004 13529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199: Generated new TSK 0
I20260928 06:45:36.399724 13529 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.7.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.1.keytab.
Entry for principal oryx/127.7.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.1.keytab.
I20260928 06:45:36.439236 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:34479
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.1.keytab
--principal=oryx/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:44951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:36.558913 13540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:36.559176 13540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:36.559218 13540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:36.563206 13540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:36.563261 13540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:36.563318 13540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:45:36.568945 13540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:44951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.1.keytab
--principal=oryx/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:34479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:36.569945 13540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:36.570940 13540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:36.580484 13545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:36.580478 13546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:36.580488 13548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:36.580677 13540 server_base.cc:1034] running on GCE node
I20260928 06:45:36.581211 13540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:36.581852 13540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:36.583009 13540 hybrid_clock.cc:648] HybridClock initialized: now 1790577936582989 us; error 31 us; skew 500 ppm
Sep 28 06:45:36 dist-test-slave-twl6 krb5kdc[13433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577936, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:36.586094 13540 init.cc:377] Logged in from keytab as oryx/127.7.48.1@KRBTEST.COM (short username oryx)
I20260928 06:45:36.586915 13540 webserver.cc:460] Webserver started at http://127.7.48.1:39793/ using document root <none> and password file <none>
I20260928 06:45:36.587430 13540 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:36.587492 13540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:36.587675 13540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:36.589751 13540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "99dc5bf901764934b1e36eeeb2d1b276"
format_stamp: "Formatted at 2026-09-28 06:45:36 on dist-test-slave-twl6"
server_key: "f50393b1c9e87b0e7e69581a37adc76d"
server_key_iv: "c130cba8c2975e0d5b201d8d45069e1d"
server_key_version: "encryptionkey@0"
I20260928 06:45:36.590267 13540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "99dc5bf901764934b1e36eeeb2d1b276"
format_stamp: "Formatted at 2026-09-28 06:45:36 on dist-test-slave-twl6"
server_key: "f50393b1c9e87b0e7e69581a37adc76d"
server_key_iv: "c130cba8c2975e0d5b201d8d45069e1d"
server_key_version: "encryptionkey@0"
I20260928 06:45:36.593907 13540 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 06:45:36.596251 13555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:36.597414 13540 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:45:36.597555 13540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "99dc5bf901764934b1e36eeeb2d1b276"
format_stamp: "Formatted at 2026-09-28 06:45:36 on dist-test-slave-twl6"
server_key: "f50393b1c9e87b0e7e69581a37adc76d"
server_key_iv: "c130cba8c2975e0d5b201d8d45069e1d"
server_key_version: "encryptionkey@0"
I20260928 06:45:36.597672 13540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:36.614543 13540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:36.617715 13540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:36.617933 13540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:36.618571 13540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:36.619426 13540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:36.619496 13540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:36.619571 13540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:36.619625 13540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:36.628811 13540 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:35937
I20260928 06:45:36.628842 13668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:35937 every 8 connection(s)
I20260928 06:45:36.629729 13540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:45:36.636406 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 13540
I20260928 06:45:36.636521 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
I20260928 06:45:36.636813 7360 external_mini_cluster.cc:1428] Setting key df29b99be3c25124544372301d87ed47
Sep 28 06:45:36 dist-test-slave-twl6 krb5kdc[13433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577936, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM
I20260928 06:45:36.644688 13517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:36.631517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:33787 (local address 127.7.48.62:34479)
0928 06:45:36.631675 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:36.631679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:36.632560 (+ 881us) server_negotiation.cc:407] Connection header received
0928 06:45:36.633877 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:36.633880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:36.633937 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:36.634038 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:36.635202 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:36.635737 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:36.637301 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:36.637650 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:36.637765 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:36.640720 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:36.640726 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:36.640730 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:36.640763 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:36.642273 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:36.642933 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:36.642936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:36.642937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:36.643013 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:36.643426 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:36.643428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:36.643429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:36.643617 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:36.643690 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:36.644436 (+ 746us) server_negotiation.cc:299] Negotiation successful
0928 06:45:36.644562 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:45:36.645996 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:36.631944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.1:33787)
0928 06:45:36.632400 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:36.632441 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:36.633667 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:36.634207 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:36.634214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:36.634567 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:36.635049 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:36.635061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:36.635862 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:36.635865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:36.637155 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:36.637168 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:36.637803 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:36.637807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:36.638008 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:36.638657 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:36.638679 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:36.640534 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:36.642396 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:36.642405 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:36.642425 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:36.642822 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:36.643105 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:36.643111 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:36.643116 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:36.643324 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:36.643723 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:36.643732 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:36.644142 (+ 410us) client_negotiation.cc:769] Sending connection context
0928 06:45:36.644388 (+ 246us) client_negotiation.cc:240] Negotiation successful
0928 06:45:36.644777 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for oryx/127.7.48.2@KRBTEST.COM; defaulting to no policy
I20260928 06:45:36.647292 13669 heartbeater.cc:347] Connected to a master server at 127.7.48.62:34479
I20260928 06:45:36.647538 13669 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:36.648114 13669 heartbeater.cc:511] Master 127.7.48.62:34479 requested a full tablet report, sending...
I20260928 06:45:36.649775 13478 ts_manager.cc:194] Registered new tserver with Master: 99dc5bf901764934b1e36eeeb2d1b276 (127.7.48.1:35937)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.48.2@KRBTEST.COM" created.
I20260928 06:45:36.650992 13478 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.48.1@KRBTEST.COM'} at 127.7.48.1:33787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.2.keytab.
Entry for principal oryx/127.7.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.2.keytab.
I20260928 06:45:36.667284 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:34479
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.2.keytab
--principal=oryx/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:44951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:36.780309 13674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:36.780575 13674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:36.780619 13674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:36.784837 13674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:36.784895 13674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:36.784988 13674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:45:36.790242 13674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:44951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.2.keytab
--principal=oryx/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:34479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:36.791402 13674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:36.792505 13674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:36.802418 13680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:36.802410 13679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:36.802511 13674 server_base.cc:1034] running on GCE node
W20260928 06:45:36.802410 13682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:36.803200 13674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:36.803861 13674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:36.805073 13674 hybrid_clock.cc:648] HybridClock initialized: now 1790577936805058 us; error 29 us; skew 500 ppm
Sep 28 06:45:36 dist-test-slave-twl6 krb5kdc[13433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577936, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:36.807845 13674 init.cc:377] Logged in from keytab as oryx/127.7.48.2@KRBTEST.COM (short username oryx)
I20260928 06:45:36.808833 13674 webserver.cc:460] Webserver started at http://127.7.48.2:36247/ using document root <none> and password file <none>
I20260928 06:45:36.809365 13674 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:36.809423 13674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:36.809600 13674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:36.811640 13674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "3b0b1b727b634600a687052b21a952b2"
format_stamp: "Formatted at 2026-09-28 06:45:36 on dist-test-slave-twl6"
server_key: "298ec7ab345119ffd429a16183c132ea"
server_key_iv: "18cb796e7ad6f9f853fdbe901e65d459"
server_key_version: "encryptionkey@0"
I20260928 06:45:36.812222 13674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "3b0b1b727b634600a687052b21a952b2"
format_stamp: "Formatted at 2026-09-28 06:45:36 on dist-test-slave-twl6"
server_key: "298ec7ab345119ffd429a16183c132ea"
server_key_iv: "18cb796e7ad6f9f853fdbe901e65d459"
server_key_version: "encryptionkey@0"
I20260928 06:45:36.815924 13674 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:45:36.818311 13689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:36.819532 13674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:36.819638 13674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "3b0b1b727b634600a687052b21a952b2"
format_stamp: "Formatted at 2026-09-28 06:45:36 on dist-test-slave-twl6"
server_key: "298ec7ab345119ffd429a16183c132ea"
server_key_iv: "18cb796e7ad6f9f853fdbe901e65d459"
server_key_version: "encryptionkey@0"
I20260928 06:45:36.819721 13674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:36.844453 13674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:36.847600 13674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:36.847766 13674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:36.848388 13674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:36.849213 13674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:36.849260 13674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:36.849303 13674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:36.849318 13674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:36.858596 13674 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:43311
I20260928 06:45:36.858606 13802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:43311 every 8 connection(s)
I20260928 06:45:36.859493 13674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:45:36.864180 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 13674
I20260928 06:45:36.864317 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:45:36.864694 7360 external_mini_cluster.cc:1428] Setting key 03a4ed811e7b33d5fe038b4ba9eb18c0
Sep 28 06:45:36 dist-test-slave-twl6 krb5kdc[13433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577936, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.7.48.3@KRBTEST.COM; defaulting to no policy
I20260928 06:45:36.875200 13517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:36.861384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:41215 (local address 127.7.48.62:34479)
0928 06:45:36.861557 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:36.861562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:36.862402 (+ 840us) server_negotiation.cc:407] Connection header received
0928 06:45:36.863756 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:36.863760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:36.863818 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:36.863956 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:36.865371 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:36.865967 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:36.867716 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:36.868128 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:36.868246 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:36.871299 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:36.871302 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:36.871304 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:36.871328 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:36.872877 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:36.873517 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:36.873519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:36.873521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:36.873601 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:36.874041 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:36.874043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:36.874044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:36.874215 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:36.874282 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:36.874968 (+ 686us) server_negotiation.cc:299] Negotiation successful
0928 06:45:36.875075 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 06:45:36.876313 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:36.861798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.2:41215)
0928 06:45:36.862247 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:36.862292 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:36.863481 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:36.864147 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:36.864154 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:36.864606 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:36.865229 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:36.865253 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:36.866140 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:36.866144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:36.867555 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:36.867569 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:36.868276 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:36.868279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:36.868431 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:36.869060 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:36.869083 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:36.871087 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:36.873009 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:36.873019 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:36.873038 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:36.873398 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:36.873712 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:36.873718 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:36.873723 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:36.873923 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:36.874342 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:36.874349 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:36.874688 (+ 339us) client_negotiation.cc:769] Sending connection context
0928 06:45:36.874932 (+ 244us) client_negotiation.cc:240] Negotiation successful
0928 06:45:36.875252 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":121,"threads_started":1}
I20260928 06:45:36.877547 13803 heartbeater.cc:347] Connected to a master server at 127.7.48.62:34479
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.48.3@KRBTEST.COM" created.
I20260928 06:45:36.877832 13803 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:36.878368 13803 heartbeater.cc:511] Master 127.7.48.62:34479 requested a full tablet report, sending...
I20260928 06:45:36.879586 13478 ts_manager.cc:194] Registered new tserver with Master: 3b0b1b727b634600a687052b21a952b2 (127.7.48.2:43311)
I20260928 06:45:36.880236 13478 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.48.2@KRBTEST.COM'} at 127.7.48.2:41215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.3.keytab.
Entry for principal oryx/127.7.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.3.keytab.
I20260928 06:45:36.894739 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:34479
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.3.keytab
--principal=oryx/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:44951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:37.007236 13808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:37.007529 13808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:37.007604 13808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:37.011862 13808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:37.012001 13808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:37.012094 13808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:45:37.017436 13808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:44951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.3.keytab
--principal=oryx/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:34479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:37.018556 13808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:37.019654 13808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:37.029790 13813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:37.029796 13814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:37.029939 13808 server_base.cc:1034] running on GCE node
W20260928 06:45:37.030121 13816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:37.030542 13808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:37.031219 13808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:37.032398 13808 hybrid_clock.cc:648] HybridClock initialized: now 1790577937032362 us; error 48 us; skew 500 ppm
Sep 28 06:45:37 dist-test-slave-twl6 krb5kdc[13433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577937, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:37.035132 13808 init.cc:377] Logged in from keytab as oryx/127.7.48.3@KRBTEST.COM (short username oryx)
I20260928 06:45:37.036157 13808 webserver.cc:460] Webserver started at http://127.7.48.3:41257/ using document root <none> and password file <none>
I20260928 06:45:37.036751 13808 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:37.036842 13808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:37.037093 13808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:37.039188 13808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6"
format_stamp: "Formatted at 2026-09-28 06:45:37 on dist-test-slave-twl6"
server_key: "8ea3148451a07e48626a6f4b6327813c"
server_key_iv: "184ccfc019646f9a4a27aae6dbac4d4f"
server_key_version: "encryptionkey@0"
I20260928 06:45:37.039775 13808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6"
format_stamp: "Formatted at 2026-09-28 06:45:37 on dist-test-slave-twl6"
server_key: "8ea3148451a07e48626a6f4b6327813c"
server_key_iv: "184ccfc019646f9a4a27aae6dbac4d4f"
server_key_version: "encryptionkey@0"
I20260928 06:45:37.043728 13808 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 06:45:37.046195 13823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:37.047432 13808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:37.047572 13808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6"
format_stamp: "Formatted at 2026-09-28 06:45:37 on dist-test-slave-twl6"
server_key: "8ea3148451a07e48626a6f4b6327813c"
server_key_iv: "184ccfc019646f9a4a27aae6dbac4d4f"
server_key_version: "encryptionkey@0"
I20260928 06:45:37.047685 13808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:37.073894 13808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:37.076936 13808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:37.077157 13808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:37.077812 13808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:37.078696 13808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:37.078768 13808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:37.078841 13808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:37.078884 13808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:37.088541 13936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:41911 every 8 connection(s)
I20260928 06:45:37.088603 13808 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:41911
I20260928 06:45:37.089598 13808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:45:37.092001 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 13808
I20260928 06:45:37.092118 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:45:37.092450 7360 external_mini_cluster.cc:1428] Setting key a4893eae7b8a546248404561490dab16
Sep 28 06:45:37 dist-test-slave-twl6 krb5kdc[13433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577937, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM
I20260928 06:45:37.104164 13517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.091520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:43417 (local address 127.7.48.62:34479)
0928 06:45:37.091676 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:37.091681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:37.092783 (+ 1102us) server_negotiation.cc:407] Connection header received
0928 06:45:37.094533 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:37.094539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:37.094599 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:37.094679 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:37.095924 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.096527 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.097699 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.097931 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.098042 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.100554 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:37.100558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:37.100561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:37.100589 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:37.101997 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.102539 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.102543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.102545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.102633 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.102988 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.102991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.102993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.103192 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:37.103299 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:37.103822 (+ 523us) server_negotiation.cc:299] Negotiation successful
0928 06:45:37.103975 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 06:45:37.104986 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.091997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.3:43417)
0928 06:45:37.092574 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:37.092646 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:37.094283 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:37.094882 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:37.094889 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:37.095254 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:37.095756 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.095769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.096657 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.096661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.097562 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.097571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.098080 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.098084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.098272 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.098896 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:37.098912 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:37.100394 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:37.102132 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.102138 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.102155 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.102415 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.102735 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.102737 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.102739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.102882 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.103297 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:37.103304 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:37.103563 (+ 259us) client_negotiation.cc:769] Sending connection context
0928 06:45:37.103781 (+ 218us) client_negotiation.cc:240] Negotiation successful
0928 06:45:37.104137 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":159,"threads_started":1}
I20260928 06:45:37.106300 13937 heartbeater.cc:347] Connected to a master server at 127.7.48.62:34479
I20260928 06:45:37.106562 13937 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:37.107091 13937 heartbeater.cc:511] Master 127.7.48.62:34479 requested a full tablet report, sending...
I20260928 06:45:37.108134 13477 ts_manager.cc:194] Registered new tserver with Master: f84f119cbe8f49f5b7d2c0fd0009a3b6 (127.7.48.3:41911)
I20260928 06:45:37.108731 13477 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.48.3@KRBTEST.COM'} at 127.7.48.3:43417
I20260928 06:45:37.118621 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.7.48.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.48.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.48.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.61.keytab.
Entry for principal oryx/127.7.48.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.61.keytab.
I20260928 06:45:37.147293 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.61:33091
--webserver_interface=127.7.48.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.61.keytab
--principal=oryx/127.7.48.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:44951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:34479,127.7.48.61:33091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:37.260020 13942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:37.260429 13942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:37.260545 13942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:37.266058 13942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:45:37.266151 13942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:37.266185 13942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:37.266227 13942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:45:37.266263 13942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:45:37.271818 13942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:44951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:34479,127.7.48.61:33091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.61.keytab
--principal=oryx/127.7.48.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.61:33091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.48.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
W20260928 06:45:37.272149 13942 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.7.48.62:34479,127.7.48.61:33091'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 06:45:37.273159 13942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:37.274235 13942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:37.282891 13948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:37.282907 13947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:37.282891 13950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:37.283226 13942 server_base.cc:1034] running on GCE node
I20260928 06:45:37.283859 13942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:37.285122 13942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:37.286320 13942 hybrid_clock.cc:648] HybridClock initialized: now 1790577937286308 us; error 32 us; skew 500 ppm
Sep 28 06:45:37 dist-test-slave-twl6 krb5kdc[13433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577937, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:37.289237 13942 init.cc:377] Logged in from keytab as oryx/127.7.48.61@KRBTEST.COM (short username oryx)
I20260928 06:45:37.290134 13942 webserver.cc:460] Webserver started at http://127.7.48.61:36497/ using document root <none> and password file <none>
I20260928 06:45:37.290706 13942 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:37.290798 13942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:37.291036 13942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:37.293182 13942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/data/instance:
uuid: "3c87976051a34263b3c105db63fb0fd0"
format_stamp: "Formatted at 2026-09-28 06:45:37 on dist-test-slave-twl6"
server_key: "7d6f7ef71471fd344f62fb403e9205b9"
server_key_iv: "737b9db4e46aa05d1524c7885b6f195f"
server_key_version: "encryptionkey@0"
I20260928 06:45:37.293779 13942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/wal/instance:
uuid: "3c87976051a34263b3c105db63fb0fd0"
format_stamp: "Formatted at 2026-09-28 06:45:37 on dist-test-slave-twl6"
server_key: "7d6f7ef71471fd344f62fb403e9205b9"
server_key_iv: "737b9db4e46aa05d1524c7885b6f195f"
server_key_version: "encryptionkey@0"
I20260928 06:45:37.297564 13942 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:45:37.300032 13957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:37.301465 13942 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:45:37.301604 13942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/wal
uuid: "3c87976051a34263b3c105db63fb0fd0"
format_stamp: "Formatted at 2026-09-28 06:45:37 on dist-test-slave-twl6"
server_key: "7d6f7ef71471fd344f62fb403e9205b9"
server_key_iv: "737b9db4e46aa05d1524c7885b6f195f"
server_key_version: "encryptionkey@0"
I20260928 06:45:37.301703 13942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:37.322227 13942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:37.325428 13942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:37.325634 13942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:37.332407 13942 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.61:33091
I20260928 06:45:37.332711 14009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.61:33091 every 8 connection(s)
I20260928 06:45:37.333413 13942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/data/info.pb
I20260928 06:45:37.333774 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 13942
I20260928 06:45:37.336588 14010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:37.341137 14010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:45:37 dist-test-slave-twl6 krb5kdc[13433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.48.61@KRBTEST.COM
Sep 28 06:45:37 dist-test-slave-twl6 krb5kdc[13433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577937, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.61@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM
I20260928 06:45:37.350325 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.335358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.7.48.61:33091)
0928 06:45:37.335779 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:37.335796 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:37.335830 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:45:37.336527 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:37.336542 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:37.336926 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:37.337207 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:37.338126 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.339110 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.340398 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.341091 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.341255 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.344454 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:37.344460 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:37.344476 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:37.344506 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:37.346575 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.347105 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.347111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.347134 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.347268 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.347593 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.347596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.347599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.348131 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:37.348862 (+ 731us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:37.349031 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 06:45:37.349363 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":143,"threads_started":1}
I20260928 06:45:37.354315 14014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.343256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.0.0.1:41918)
0928 06:45:37.344174 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:37.344228 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:37.344359 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:37.344653 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:37.344658 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:37.345158 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:37.345279 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.345285 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.346163 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.346168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.346984 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.346994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.347578 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.347582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.347698 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.348318 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:37.348333 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:37.349970 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:37.352818 (+ 2848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.352823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.352827 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.353047 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.353299 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.353301 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.353302 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.353374 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.353821 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:37.353825 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:37.353918 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 06:45:37.354034 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 06:45:37.354168 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"spinlock_wait_cycles":896,"thread_start_us":136,"threads_started":1}
I20260928 06:45:37.354369 13517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.343283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41918 (local address 127.7.48.62:34479)
0928 06:45:37.343461 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:37.343466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:37.344313 (+ 847us) server_negotiation.cc:407] Connection header received
0928 06:45:37.344474 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:37.344477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:37.344531 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:37.344620 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:37.345405 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.346040 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.347162 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.347456 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.347568 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.350113 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:37.350117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:37.350120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:37.350147 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:37.352718 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.353134 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.353137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.353139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.353218 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.353486 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.353489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.353490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.353726 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:37.353819 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:37.354129 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 06:45:37.354254 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 06:45:37.355496 14010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:45:37 dist-test-slave-twl6 krb5kdc[13433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577937, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.61@KRBTEST.COM for oryx/127.7.48.61@KRBTEST.COM
I20260928 06:45:37.363659 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.356050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49360 (local address 127.7.48.61:33091)
0928 06:45:37.356155 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:37.356160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:37.356236 (+ 76us) server_negotiation.cc:407] Connection header received
0928 06:45:37.356373 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:37.356376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:37.356416 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:37.356499 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:37.356870 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.357361 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.358076 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.358308 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.358414 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.360782 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:37.360786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:37.360788 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:37.360819 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:37.361986 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.362386 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.362392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.362394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.362467 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.362739 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.362743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.362745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.362982 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:37.363091 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:37.363328 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 06:45:37.363474 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":42}
I20260928 06:45:37.363659 14014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.356010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:33091 (local address 127.0.0.1:49360)
0928 06:45:37.356155 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:37.356170 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:37.356274 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:37.356511 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:37.356514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:37.356675 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:37.356753 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.356758 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.357482 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.357485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.357966 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.357972 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.358412 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.358415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.358543 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.358955 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:37.358966 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:37.360645 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:37.362097 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.362103 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.362105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.362290 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.362565 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.362570 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.362572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.362650 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.363079 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:37.363082 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:37.363173 (+ 91us) client_negotiation.cc:769] Sending connection context
0928 06:45:37.363283 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 06:45:37.363423 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":3}
I20260928 06:45:37.366628 14010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0: Bootstrap starting.
I20260928 06:45:37.368952 14010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:37.369640 14010 log.cc:824] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:37.371531 14010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0: No bootstrap required, opened a new log
I20260928 06:45:37.374536 14010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } }
I20260928 06:45:37.374730 14010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:37.374771 14010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c87976051a34263b3c105db63fb0fd0, State: Initialized, Role: FOLLOWER
I20260928 06:45:37.375149 14010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } }
I20260928 06:45:37.375557 14016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } } }
I20260928 06:45:37.375706 14016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:45:37.376344 14010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 06:45:37.380942 14027 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:45:37.381028 14027 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:45:37.383226 13942 master_runner.cc:428] Detected that this master 3c87976051a34263b3c105db63fb0fd0 is joining an existing cluster
I20260928 06:45:37.383296 13942 master_runner.cc:432] Initiating AddMaster RPC to add 127.7.48.61:33091
I20260928 06:45:37.393968 13517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.385870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41926 (local address 127.7.48.62:34479)
0928 06:45:37.385996 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:37.386001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:37.386015 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:45:37.386057 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:37.386060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:37.386108 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:37.386202 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:37.386930 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.387769 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.388931 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.389153 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.389266 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.390683 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:37.390687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:37.390691 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:37.390722 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:37.392275 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.392766 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.392770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.392772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.392864 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.393137 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.393141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.393143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.393323 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:37.393439 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:37.393712 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 06:45:37.393819 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 06:45:37.393989 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.385461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.0.0.1:41926)
0928 06:45:37.385799 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:37.385817 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:37.385917 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:37.386224 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:37.386227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:37.386517 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:37.386798 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.386803 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.387915 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.387918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.388815 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.388823 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.389267 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.389271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.389367 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.389862 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:37.389877 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:37.390254 (+ 377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:37.392394 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.392397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.392399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.392661 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.392966 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.392968 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.392970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.393039 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.393440 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:37.393444 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:37.393539 (+ 95us) client_negotiation.cc:769] Sending connection context
0928 06:45:37.393648 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 06:45:37.393786 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":90,"thread_start_us":144,"threads_started":1}
I20260928 06:45:37.394419 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.385513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:33091 (local address 127.0.0.1:49362)
0928 06:45:37.385898 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:37.385909 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:37.386014 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:37.386312 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:37.386315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:37.386519 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:37.386795 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.386803 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.387482 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.387484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.388268 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.388278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.389055 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.389058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.389147 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.389709 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:37.389725 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:37.390244 (+ 519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:37.392206 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.392209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.392211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.392436 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.392985 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.392989 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.392991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.393083 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.393517 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:37.393519 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:37.393603 (+ 84us) client_negotiation.cc:769] Sending connection context
0928 06:45:37.394145 (+ 542us) client_negotiation.cc:240] Negotiation successful
0928 06:45:37.394267 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":89,"thread_start_us":145,"threads_started":1}
I20260928 06:45:37.394025 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.385407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49362 (local address 127.7.48.61:33091)
0928 06:45:37.385534 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:37.385541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:37.385969 (+ 428us) server_negotiation.cc:407] Connection header received
0928 06:45:37.386082 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:37.386087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:37.386133 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:37.386211 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:37.386914 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.387398 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.388380 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.388819 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.388928 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.390377 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:37.390381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:37.390383 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:37.390412 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:37.392096 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.392582 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.392588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.392593 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.392667 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.393182 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.393185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.393186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.393372 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:37.393449 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:37.393753 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 06:45:37.393861 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":49}
Sep 28 06:45:37 dist-test-slave-twl6 krb5kdc[13433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577936, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.62@KRBTEST.COM for oryx/127.7.48.61@KRBTEST.COM
I20260928 06:45:37.409130 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.399225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49364 (local address 127.7.48.61:33091)
0928 06:45:37.399583 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:37.399587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:37.399988 (+ 401us) server_negotiation.cc:407] Connection header received
0928 06:45:37.400178 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:37.400180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:37.400222 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:37.400287 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:37.401076 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.401543 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.402781 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.403071 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.403179 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.405947 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:37.405951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:37.405953 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:37.405984 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:37.407340 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.407809 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.407813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.407815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.407942 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.408236 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.408240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.408242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.408414 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:37.408519 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:37.408858 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 06:45:37.408971 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260928 06:45:37.409174 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.399213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:33091 (local address 127.0.0.1:49364)
0928 06:45:37.399835 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:37.399889 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:37.400053 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:37.400344 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:37.400350 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:37.400805 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:37.400947 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.400955 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.401654 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.401658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.402673 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.402684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.403191 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.403195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.403321 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.403895 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:37.403913 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:37.405757 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:37.407451 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.407462 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.407469 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.407706 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.408041 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.408044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.408047 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.408144 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.408524 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:37.408533 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:37.408658 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 06:45:37.408805 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 06:45:37.409000 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":103,"threads_started":1}
I20260928 06:45:37.410652 13477 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.7.48.61:33091: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3c87976051a34263b3c105db63fb0fd0"
member_type: NON_VOTER
last_known_addr {
host: "127.7.48.61"
port: 33091
}
attrs {
promote: true
}
}
dest_uuid: "1137a312ab134aa4afef56800ac54199"
cas_config_opid_index: -1
I20260928 06:45:37.411456 13477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: NON_VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: true } }
I20260928 06:45:37.413657 14039 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3c87976051a34263b3c105db63fb0fd0 (127.7.48.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: NON_VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: true } } }
I20260928 06:45:37.414019 13984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:37.415442 13984 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 1 FOLLOWER]: Refusing update from remote peer 1137a312ab134aa4afef56800ac54199: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 06:45:37.415486 14039 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.7.48.61:33091
I20260928 06:45:37.415941 13942 master.cc:537] Master@127.7.48.61:33091 shutting down...
WARNING: no policy specified for oryx/127.7.48.60@KRBTEST.COM; defaulting to no policy
I20260928 06:45:37.417086 14039 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: NON_VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:37.418883 14038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1137a312ab134aa4afef56800ac54199" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: NON_VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: true } } }
I20260928 06:45:37.418998 14038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.48.60@KRBTEST.COM" created.
W20260928 06:45:37.420179 13464 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 06:45:37.421132 13464 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 -> Peer 3c87976051a34263b3c105db63fb0fd0 (127.7.48.61:33091): Couldn't send request to peer 3c87976051a34263b3c105db63fb0fd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 06:45:37.421682 13942 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:37.421996 13942 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:37.422112 13942 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0: stopping tablet replica
I20260928 06:45:37.437167 13942 master.cc:552] Master@127.7.48.61:33091 shutdown complete.
I20260928 06:45:37.437255 13942 master_runner.cc:305] Clearing existing system tablet
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.48.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.60.keytab.
Entry for principal oryx/127.7.48.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.60.keytab.
I20260928 06:45:37.437534 13942 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:37.437608 13942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:37.438220 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.60:40403
--webserver_interface=127.7.48.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.60.keytab
--principal=oryx/127.7.48.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:44951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:34479,127.7.48.61:33091,127.7.48.60:40403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:45:37.440268 13942 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:45:37.441227 14046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:37.441506 13942 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:37.441632 13942 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:45:37.441666 13942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/wal
uuid: "3c87976051a34263b3c105db63fb0fd0"
format_stamp: "Formatted at 2026-09-28 06:45:37 on dist-test-slave-twl6"
server_key: "7d6f7ef71471fd344f62fb403e9205b9"
server_key_iv: "737b9db4e46aa05d1524c7885b6f195f"
server_key_version: "encryptionkey@0"
I20260928 06:45:37.443984 13942 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:37.446385 13942 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 06:45:37.446506 13942 log.cc:1196] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 06:45:37.446777 13942 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0: Deleting consensus metadata
I20260928 06:45:37.447031 13942 master_runner.cc:315] Copying system tablet from 127.7.48.62:34479
I20260928 06:45:37.448233 13942 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0: tablet copy: Beginning tablet copy session from remote peer at address 127.7.48.62:34479
I20260928 06:45:37.457862 13517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.448952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41932 (local address 127.7.48.62:34479)
0928 06:45:37.449083 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:37.449089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:37.449372 (+ 283us) server_negotiation.cc:407] Connection header received
0928 06:45:37.449512 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:37.449515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:37.449567 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:37.449664 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:37.450222 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.451048 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.452413 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.452787 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.452898 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.454279 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:37.454283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:37.454287 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:37.454317 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:37.456079 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.456581 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.456584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.456586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.456669 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.456971 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.456974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.456976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.457197 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:37.457297 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:37.457583 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 06:45:37.457699 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:45:37.457897 14052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.448860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.0.0.1:41932)
0928 06:45:37.449249 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:37.449268 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:37.449407 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:37.449698 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:37.449701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:37.449949 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:37.450090 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.450097 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.451202 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.451205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.452262 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.452273 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.452911 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.452914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.453010 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.453677 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:37.453691 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:37.454127 (+ 436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:37.456201 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.456205 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.456207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.456485 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.456761 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.456764 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.456766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.456871 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.457296 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:37.457299 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:37.457399 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 06:45:37.457506 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 06:45:37.457653 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":93,"threads_started":1}
I20260928 06:45:37.458745 13498 tablet_copy_service.cc:140] P 1137a312ab134aa4afef56800ac54199: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3c87976051a34263b3c105db63fb0fd0 ({username='oryx', principal='oryx/127.7.48.61@KRBTEST.COM'} at 127.0.0.1:41932)
I20260928 06:45:37.458926 13498 tablet_copy_service.cc:161] P 1137a312ab134aa4afef56800ac54199: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3c87976051a34263b3c105db63fb0fd0 at {username='oryx', principal='oryx/127.7.48.61@KRBTEST.COM'} at 127.0.0.1:41932: session id = 3c87976051a34263b3c105db63fb0fd0-00000000000000000000000000000000
I20260928 06:45:37.461016 13498 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 06:45:37.462687 13942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 06:45:37.465301 13942 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0: tablet copy: Starting download of 0 data blocks...
I20260928 06:45:37.465576 13942 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0: tablet copy: Starting download of 1 WAL segments...
I20260928 06:45:37.467417 13942 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 06:45:37.469417 13498 tablet_copy_service.cc:342] P 1137a312ab134aa4afef56800ac54199: Request end of tablet copy session 3c87976051a34263b3c105db63fb0fd0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.7.48.61@KRBTEST.COM'} at 127.0.0.1:41932
I20260928 06:45:37.469545 13498 tablet_copy_service.cc:433] P 1137a312ab134aa4afef56800ac54199: ending tablet copy session 3c87976051a34263b3c105db63fb0fd0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3c87976051a34263b3c105db63fb0fd0
I20260928 06:45:37.477097 13942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:37.478914 14056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:37.478965 14053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:37.479225 13942 server_base.cc:1034] running on GCE node
W20260928 06:45:37.479231 14054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:37.479578 13942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:37.479919 13942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:37.481060 13942 hybrid_clock.cc:648] HybridClock initialized: now 1790577937481048 us; error 28 us; skew 500 ppm
Sep 28 06:45:37 dist-test-slave-twl6 krb5kdc[13433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577937, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:37.482805 13942 init.cc:377] Logged in from keytab as oryx/127.7.48.61@KRBTEST.COM (short username oryx)
I20260928 06:45:37.483340 13942 webserver.cc:460] Webserver started at http://127.7.48.61:46351/ using document root <none> and password file <none>
I20260928 06:45:37.483527 13942 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:37.483574 13942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:37.485004 13942 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:45:37.485966 14063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:37.486246 13942 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:45:37.486320 13942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/wal
uuid: "3c87976051a34263b3c105db63fb0fd0"
format_stamp: "Formatted at 2026-09-28 06:45:37 on dist-test-slave-twl6"
server_key: "7d6f7ef71471fd344f62fb403e9205b9"
server_key_iv: "737b9db4e46aa05d1524c7885b6f195f"
server_key_version: "encryptionkey@0"
I20260928 06:45:37.486397 13942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:37.496491 13942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:37.498569 13942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:37.503463 13942 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.61:33091
I20260928 06:45:37.504177 13942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-1/data/info.pb
I20260928 06:45:37.505970 14116 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:45:37.506991 14116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0: Bootstrap starting.
I20260928 06:45:37.508100 14115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.61:33091 every 8 connection(s)
I20260928 06:45:37.517614 14116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:45:37.517930 14116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0: Bootstrap complete.
I20260928 06:45:37.518554 14116 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: NON_VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: true } }
I20260928 06:45:37.518692 14116 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3c87976051a34263b3c105db63fb0fd0, State: Initialized, Role: LEARNER
I20260928 06:45:37.518792 14116 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: NON_VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: true } }
I20260928 06:45:37.519153 14119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: NON_VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: true } } }
I20260928 06:45:37.519251 14116 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:45:37.519357 14119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [sys.catalog]: This master's current role is: LEARNER
I20260928 06:45:37.522912 14130 catalog_manager.cc:1259] Loaded cluster ID: 5736c5ca81164407b939a2e06709be0c
I20260928 06:45:37.522970 14130 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0: loading cluster ID for follower catalog manager: success
I20260928 06:45:37.525204 14130 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0: acquiring CA information for follower catalog manager: success
I20260928 06:45:37.526650 14130 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 06:45:37.567793 14045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:37.568071 14045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:37.568166 14045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:37.572219 14045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:45:37.572299 14045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:37.572316 14045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:37.572376 14045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:45:37.572402 14045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:45:37.577837 14045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:44951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:34479,127.7.48.61:33091,127.7.48.60:40403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.60.keytab
--principal=oryx/127.7.48.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.60:40403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.48.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:37.578925 14045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:37.580066 14045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:37.588779 14135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:37.588779 14136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:37.588784 14138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:37.589119 14045 server_base.cc:1034] running on GCE node
I20260928 06:45:37.589670 14045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:37.590804 14045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:37.591941 14045 hybrid_clock.cc:648] HybridClock initialized: now 1790577937591926 us; error 39 us; skew 500 ppm
Sep 28 06:45:37 dist-test-slave-twl6 krb5kdc[13433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577937, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:37.594681 14045 init.cc:377] Logged in from keytab as oryx/127.7.48.60@KRBTEST.COM (short username oryx)
I20260928 06:45:37.595554 14045 webserver.cc:460] Webserver started at http://127.7.48.60:46355/ using document root <none> and password file <none>
I20260928 06:45:37.596168 14045 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:37.596254 14045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:37.596477 14045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:37.598439 14045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/data/instance:
uuid: "3ed6876b7d43478da762015583cd0eeb"
format_stamp: "Formatted at 2026-09-28 06:45:37 on dist-test-slave-twl6"
server_key: "4638855de4c45a5a10e2f928f5b857fb"
server_key_iv: "6ea02022373b2ac4e9cc8ca7737c4ad9"
server_key_version: "encryptionkey@0"
I20260928 06:45:37.598984 14045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/wal/instance:
uuid: "3ed6876b7d43478da762015583cd0eeb"
format_stamp: "Formatted at 2026-09-28 06:45:37 on dist-test-slave-twl6"
server_key: "4638855de4c45a5a10e2f928f5b857fb"
server_key_iv: "6ea02022373b2ac4e9cc8ca7737c4ad9"
server_key_version: "encryptionkey@0"
I20260928 06:45:37.602742 14045 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 06:45:37.605106 14145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:37.606320 14045 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:45:37.606451 14045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/wal
uuid: "3ed6876b7d43478da762015583cd0eeb"
format_stamp: "Formatted at 2026-09-28 06:45:37 on dist-test-slave-twl6"
server_key: "4638855de4c45a5a10e2f928f5b857fb"
server_key_iv: "6ea02022373b2ac4e9cc8ca7737c4ad9"
server_key_version: "encryptionkey@0"
I20260928 06:45:37.606557 14045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:37.622611 14045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:37.625959 14045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:37.626168 14045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:37.633775 14045 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.60:40403
I20260928 06:45:37.634059 14197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.60:40403 every 8 connection(s)
I20260928 06:45:37.634771 14045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/data/info.pb
I20260928 06:45:37.637231 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 14045
I20260928 06:45:37.637696 14198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:37.642053 14198 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:45:37 dist-test-slave-twl6 krb5kdc[13433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.48.60@KRBTEST.COM
Sep 28 06:45:37 dist-test-slave-twl6 krb5kdc[13433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577937, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.60@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM
I20260928 06:45:37.652518 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.638960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59510 (local address 127.7.48.60:40403)
0928 06:45:37.639364 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:37.639380 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:37.639415 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:45:37.640289 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:37.640308 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:37.640629 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:37.640974 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:37.641765 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.642792 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.643680 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.644377 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.644554 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.647089 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:37.647094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:37.647103 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:37.647131 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:37.648940 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.649585 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.649591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.649597 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.649735 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.650096 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.650099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.650100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.650679 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:37.650897 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:37.651149 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 06:45:37.651433 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":136,"threads_started":1}
I20260928 06:45:37.653173 13517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.643935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41948 (local address 127.7.48.62:34479)
0928 06:45:37.644064 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:37.644069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:37.644257 (+ 188us) server_negotiation.cc:407] Connection header received
0928 06:45:37.644418 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:37.644421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:37.644497 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:37.644587 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:37.645406 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.645925 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.646926 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.647146 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.647257 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.649590 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:37.649594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:37.649596 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:37.649623 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:37.651353 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.651808 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.651811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.651813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.651928 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.652219 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.652222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.652224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.652429 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:37.652529 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:37.652910 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 06:45:37.653021 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 06:45:37.653198 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.643717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.0.0.1:41948)
0928 06:45:37.644140 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:37.644177 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:37.644310 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:37.644635 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:37.644641 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:37.645118 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:37.645275 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.645281 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.646057 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.646060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.646801 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.646810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.647256 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.647260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.647394 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.647954 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:37.647977 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:37.649443 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:37.651468 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.651477 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.651482 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.651721 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.652044 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.652046 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.652048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.652125 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.652537 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:37.652542 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:37.652677 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 06:45:37.652835 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 06:45:37.653021 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":127,"threads_started":1}
I20260928 06:45:37.653861 13669 heartbeater.cc:503] Master 127.7.48.62:34479 was elected leader, sending a full tablet report...
I20260928 06:45:37.654263 14198 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:45:37 dist-test-slave-twl6 krb5kdc[13433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577937, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.60@KRBTEST.COM for oryx/127.7.48.61@KRBTEST.COM
I20260928 06:45:37.661947 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.654712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:33091 (local address 127.0.0.1:49370)
0928 06:45:37.655149 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:37.655183 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:37.655252 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:37.655481 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:37.655483 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:37.655601 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:37.655684 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.655688 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.656579 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.656581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.657082 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.657089 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.657492 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.657495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.657559 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.658000 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:37.658020 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:37.659159 (+ 1139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:37.660695 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.660698 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.660699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.660873 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.661108 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.661110 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.661112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.661180 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.661508 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:37.661511 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:37.661591 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 06:45:37.661690 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 06:45:37.661794 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379}
I20260928 06:45:37.661960 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.654876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49370 (local address 127.7.48.61:33091)
0928 06:45:37.655188 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:37.655202 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:37.655221 (+ 19us) server_negotiation.cc:407] Connection header received
0928 06:45:37.655334 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:37.655336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:37.655375 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:37.655449 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:37.655779 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.656489 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.657174 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.657392 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.657463 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.659284 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:37.659287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:37.659289 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:37.659311 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:37.660604 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.660961 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.660963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.660964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.661033 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.661248 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.661250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.661251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.661401 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:37.661469 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:37.661735 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 06:45:37.661829 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":95,"threads_started":1}
I20260928 06:45:37.662637 14198 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:45:37 dist-test-slave-twl6 krb5kdc[13433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577937, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.60@KRBTEST.COM for oryx/127.7.48.60@KRBTEST.COM
I20260928 06:45:37.670214 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.663120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:40403 (local address 127.0.0.1:59516)
0928 06:45:37.663235 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:37.663250 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:37.663329 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:37.663530 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:37.663533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:37.663691 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:37.663769 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.663776 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.664847 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.664853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.665348 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.665354 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.665801 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.665804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.665927 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.666295 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:37.666305 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:37.667389 (+ 1084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:37.668990 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.668992 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.668994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.669168 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.669392 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.669394 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.669395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.669460 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.669835 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:37.669837 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:37.669905 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 06:45:37.669981 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:45:37.670068 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":38}
I20260928 06:45:37.670248 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.663141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59516 (local address 127.7.48.60:40403)
0928 06:45:37.663247 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:37.663252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:37.663293 (+ 41us) server_negotiation.cc:407] Connection header received
0928 06:45:37.663396 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:37.663402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:37.663442 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:37.663521 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:37.663862 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.664735 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.665453 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.665698 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.665804 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.667513 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:37.667516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:37.667518 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:37.667537 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:37.668908 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.669245 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.669248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.669249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.669315 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.669571 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.669574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.669575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.669754 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:37.669835 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:37.670043 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 06:45:37.670129 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":43}
I20260928 06:45:37.673210 14198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb: Bootstrap starting.
I20260928 06:45:37.675571 14198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:37.676373 14198 log.cc:824] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:37.678236 14198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb: No bootstrap required, opened a new log
I20260928 06:45:37.681399 14198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } }
I20260928 06:45:37.681612 14198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:37.681648 14198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed6876b7d43478da762015583cd0eeb, State: Initialized, Role: FOLLOWER
I20260928 06:45:37.682072 14198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } }
I20260928 06:45:37.682571 14204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } } }
I20260928 06:45:37.682710 14204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:45:37.683488 14198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [sys.catalog]: configured and running, proceeding with master startup.
W20260928 06:45:37.688522 14215 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:45:37.688618 14215 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:45:37.691931 14045 master_runner.cc:428] Detected that this master 3ed6876b7d43478da762015583cd0eeb is joining an existing cluster
I20260928 06:45:37.692001 14045 master_runner.cc:432] Initiating AddMaster RPC to add 127.7.48.60:40403
I20260928 06:45:37.703913 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.695276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49374 (local address 127.7.48.61:33091)
0928 06:45:37.695516 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:37.695520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:37.695530 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:45:37.695564 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:37.695565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:37.695602 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:37.695703 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:37.696295 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.697057 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.698375 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.698730 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.698804 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.700363 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:37.700366 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:37.700369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:37.700397 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:37.702151 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.702598 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.702602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.702604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.702693 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.703003 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.703006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.703008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.703204 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:37.703316 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:37.703596 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 06:45:37.703723 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260928 06:45:37.703923 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.694534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:33091 (local address 127.0.0.1:49374)
0928 06:45:37.695298 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:37.695311 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:37.695388 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:37.695722 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:37.695725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:37.696012 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:37.696154 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.696161 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.697192 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.697195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.698230 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.698240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.698949 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.698951 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.699161 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.699783 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:37.699804 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:37.700208 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:37.702268 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.702273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.702275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.702495 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.702791 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.702794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.702796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.702904 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.703305 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:37.703308 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:37.703412 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 06:45:37.703523 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 06:45:37.703682 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":153,"spinlock_wait_cycles":3968,"thread_start_us":129,"threads_started":1}
I20260928 06:45:37.706111 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 13540
I20260928 06:45:37.713007 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:35937
--local_ip_for_outbound_sockets=127.7.48.1
--tserver_master_addrs=127.7.48.62:34479
--webserver_port=39793
--webserver_interface=127.7.48.1
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.1.keytab
--principal=oryx/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:44951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.48.62:34479,127.7.48.61:33091
--tserver_master_addrs=127.7.48.62:34479,127.7.48.61:33091,127.7.48.60:40403 with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:45:37.729904 14223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.694972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:40403 (local address 127.0.0.1:59532)
0928 06:45:37.696162 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:37.696174 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:37.696281 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:37.696553 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:37.696558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:37.696760 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:37.696881 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.696888 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.697890 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.697893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.698771 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.698781 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.699445 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.699448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.699533 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.724744 (+ 25211us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:37.724756 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:37.725506 (+ 750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:37.728022 (+ 2516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.728026 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.728029 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.728315 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.728643 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.728649 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.728653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.728733 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.729302 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:37.729304 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:37.729389 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 06:45:37.729491 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 06:45:37.729671 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1129,"mutex_wait_us":3,"spinlock_wait_cycles":2432,"thread_start_us":84,"threads_started":1}
I20260928 06:45:37.730646 13517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.694556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41964 (local address 127.7.48.62:34479)
0928 06:45:37.694703 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:37.694709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:37.694995 (+ 286us) server_negotiation.cc:407] Connection header received
0928 06:45:37.695217 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:37.695221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:37.695278 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:37.695364 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:37.696065 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.696919 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.698374 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.698714 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.698807 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.725757 (+ 26950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:37.725761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:37.725765 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:37.725799 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:37.727930 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.728708 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.728712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.728714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.728807 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.729112 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.729115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.729116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.729353 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:37.729442 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:37.730413 (+ 971us) server_negotiation.cc:299] Negotiation successful
0928 06:45:37.730535 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:45:37.731047 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.695007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59532 (local address 127.7.48.60:40403)
0928 06:45:37.695108 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:37.695112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:37.696233 (+ 1121us) server_negotiation.cc:407] Connection header received
0928 06:45:37.696369 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:37.696371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:37.696404 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:37.696447 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:37.696994 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.697735 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.698898 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.699244 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.699329 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.725735 (+ 26406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:37.725739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:37.725741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:37.725767 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:37.727839 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.728428 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.728432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.728435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.728534 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.728826 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.728829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.728833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.729172 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:37.729261 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:37.730201 (+ 940us) server_negotiation.cc:299] Negotiation successful
0928 06:45:37.730315 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":41}
I20260928 06:45:37.729951 14222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.694473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.0.0.1:41964)
0928 06:45:37.694904 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:37.694919 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:37.695070 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:37.695486 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:37.695489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:37.695740 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:37.695914 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.695924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.697072 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.697075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.698230 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.698240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.698947 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.698951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.699164 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.724662 (+ 25498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:37.724694 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:37.725506 (+ 812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:37.728062 (+ 2556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.728067 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.728071 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.728592 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.728912 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.728914 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.728916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.728986 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.729485 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:37.729488 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:37.729577 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 06:45:37.729672 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 06:45:37.729788 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":158,"spinlock_wait_cycles":7808,"thread_start_us":108,"threads_started":1}
Sep 28 06:45:37 dist-test-slave-twl6 krb5kdc[13433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577936, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.62@KRBTEST.COM for oryx/127.7.48.60@KRBTEST.COM
I20260928 06:45:37.743359 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.735609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:40403 (local address 127.0.0.1:59538)
0928 06:45:37.735721 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:37.735735 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:37.735814 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:37.736166 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:37.736169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:37.736361 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:37.736465 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.736471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.737243 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.737246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.737802 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.737810 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.738294 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.738298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.738400 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.738876 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:37.738888 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:37.740172 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:37.741853 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.741858 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.741860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.742070 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.742349 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.742353 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.742355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.742444 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.742808 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:37.742812 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:37.742919 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 06:45:37.743033 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 06:45:37.743193 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 06:45:37.743359 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.735782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59538 (local address 127.7.48.60:40403)
0928 06:45:37.735928 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:37.735932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:37.735943 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:45:37.736015 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:37.736018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:37.736058 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:37.736148 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:37.736584 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.737148 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.737927 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.738160 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.738264 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.740324 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:37.740327 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:37.740329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:37.740348 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:37.741735 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.742176 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.742185 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.742186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.742257 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.742541 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.742544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.742546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.742712 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:37.742817 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:37.743087 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 06:45:37.743192 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:45:37.744735 13477 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.7.48.60:40403: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3ed6876b7d43478da762015583cd0eeb"
member_type: NON_VOTER
last_known_addr {
host: "127.7.48.60"
port: 40403
}
attrs {
promote: true
}
}
dest_uuid: "1137a312ab134aa4afef56800ac54199"
cas_config_opid_index: 5
I20260928 06:45:37.745319 13477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: NON_VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: true } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: NON_VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: true } }
I20260928 06:45:37.746440 14040 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3ed6876b7d43478da762015583cd0eeb (127.7.48.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: NON_VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: true } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: NON_VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: true } } }
I20260928 06:45:37.748016 14040 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.7.48.60:40403
I20260928 06:45:37.748234 14040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1137a312ab134aa4afef56800ac54199" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: NON_VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: true } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: NON_VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: true } } }
I20260928 06:45:37.748456 14040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:37.749708 14172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:37.751255 14172 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 1 FOLLOWER]: Refusing update from remote peer 1137a312ab134aa4afef56800ac54199: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 06:45:37.751850 14038 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: NON_VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:37.752893 14045 master.cc:537] Master@127.7.48.60:40403 shutting down...
I20260928 06:45:37.756696 14172 raft_consensus.cc:858] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: NON_VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: NON_VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: true } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: NON_VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: true } }
I20260928 06:45:37.756765 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.748970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:33091 (local address 127.0.0.1:49390)
0928 06:45:37.749652 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:37.749667 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:37.749757 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:37.750237 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:37.750240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:37.750260 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:37.750346 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.750352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.751471 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.751474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.754225 (+ 2751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.754237 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.756249 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.756254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.756379 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.756461 (+ 82us) client_negotiation.cc:769] Sending connection context
0928 06:45:37.756547 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:45:37.756626 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610}
W20260928 06:45:37.757052 14172 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: NON_VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: NON_VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: true } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: NON_VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: true } }
I20260928 06:45:37.757186 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.749413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49390 (local address 127.7.48.61:33091)
0928 06:45:37.749556 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:37.749560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:37.749849 (+ 289us) server_negotiation.cc:407] Connection header received
0928 06:45:37.749914 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:37.749919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:37.749992 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:37.750088 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:37.750494 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.751318 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.754375 (+ 3057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.754992 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.755102 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.755265 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:37.756968 (+ 1703us) server_negotiation.cc:299] Negotiation successful
0928 06:45:37.757059 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 06:45:37.757622 14090 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 1 LEARNER]: Refusing update from remote peer 1137a312ab134aa4afef56800ac54199: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 06:45:37.758057 14038 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: NON_VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 06:45:37.758845 14172 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3c87976051a34263b3c105db63fb0fd0 (127.7.48.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: NON_VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: true } } }
I20260928 06:45:37.759935 14090 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3ed6876b7d43478da762015583cd0eeb (127.7.48.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: NON_VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: true } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: NON_VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: true } } }
I20260928 06:45:37.761740 14229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1137a312ab134aa4afef56800ac54199. Latest consensus state: current_term: 1 leader_uuid: "1137a312ab134aa4afef56800ac54199" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: NON_VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: true } } }
I20260928 06:45:37.761888 14119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1137a312ab134aa4afef56800ac54199. Latest consensus state: current_term: 1 leader_uuid: "1137a312ab134aa4afef56800ac54199" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: NON_VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: true } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: NON_VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: true } } }
I20260928 06:45:37.761996 14119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [sys.catalog]: This master's current role is: LEARNER
I20260928 06:45:37.762091 14229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 06:45:37.762084 14119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1137a312ab134aa4afef56800ac54199" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: NON_VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: true } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: NON_VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: true } } }
I20260928 06:45:37.762182 14119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [sys.catalog]: This master's current role is: LEARNER
I20260928 06:45:37.762300 14229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1137a312ab134aa4afef56800ac54199" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: NON_VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: true } } }
I20260928 06:45:37.762458 14229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 06:45:37.762593 14229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1137a312ab134aa4afef56800ac54199" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: NON_VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: true } } }
I20260928 06:45:37.762719 14229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 06:45:37.762779 14038 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199: attempting to promote NON_VOTER 3c87976051a34263b3c105db63fb0fd0 to VOTER
I20260928 06:45:37.763178 14038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: NON_VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: true } }
I20260928 06:45:37.764551 14090 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 1 LEARNER]: Refusing update from remote peer 1137a312ab134aa4afef56800ac54199: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 06:45:37.765029 14040 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260928 06:45:37.765482 13464 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 -> Peer 3ed6876b7d43478da762015583cd0eeb (127.7.48.60:40403): Couldn't send request to peer 3ed6876b7d43478da762015583cd0eeb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 06:45:37.767520 14039 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3c87976051a34263b3c105db63fb0fd0 (127.7.48.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: NON_VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: true } } }
I20260928 06:45:37.768072 14090 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3c87976051a34263b3c105db63fb0fd0 (127.7.48.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: NON_VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: true } } }
I20260928 06:45:37.769207 14039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1137a312ab134aa4afef56800ac54199" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: NON_VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: true } } }
I20260928 06:45:37.769325 14039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:37.769470 14039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1137a312ab134aa4afef56800ac54199" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: NON_VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: true } } }
I20260928 06:45:37.769559 14039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:37.769651 14119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1137a312ab134aa4afef56800ac54199" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: NON_VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: true } } }
I20260928 06:45:37.769760 14119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:45:37.774806 14045 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260928 06:45:37.774968 14225 mvcc.cc:205] Tried to move back new op lower bound from 7334207227485712384 to 7334207227342217216. Current Snapshot: MvccSnapshot[applied={T|T < 7334207227378044928}]
I20260928 06:45:37.775070 14045 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260928 06:45:37.775146 14045 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb: stopping tablet replica
I20260928 06:45:37.785465 14045 master.cc:552] Master@127.7.48.60:40403 shutdown complete.
I20260928 06:45:37.785547 14045 master_runner.cc:305] Clearing existing system tablet
I20260928 06:45:37.785784 14045 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:37.785908 14045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:37.787140 14045 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:45:37.788149 14235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:37.788375 14045 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:37.788538 14045 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:45:37.788576 14045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/wal
uuid: "3ed6876b7d43478da762015583cd0eeb"
format_stamp: "Formatted at 2026-09-28 06:45:37 on dist-test-slave-twl6"
server_key: "4638855de4c45a5a10e2f928f5b857fb"
server_key_iv: "6ea02022373b2ac4e9cc8ca7737c4ad9"
server_key_version: "encryptionkey@0"
I20260928 06:45:37.790006 14045 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:37.792181 14045 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 06:45:37.792289 14045 log.cc:1196] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 06:45:37.792622 14045 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb: Deleting consensus metadata
I20260928 06:45:37.792814 14045 master_runner.cc:315] Copying system tablet from 127.7.48.62:34479
I20260928 06:45:37.794024 14045 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb: tablet copy: Beginning tablet copy session from remote peer at address 127.7.48.62:34479
I20260928 06:45:37.802150 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.794599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.0.0.1:41978)
0928 06:45:37.794919 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:37.794944 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:37.795080 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:37.795351 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:37.795354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:37.795572 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:37.795671 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.795677 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.796526 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.796531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.797119 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.797127 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.797609 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.797612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.797706 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.798284 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:37.798297 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:37.798628 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:37.800605 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.800609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.800612 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.800822 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.801115 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.801119 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.801121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.801210 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.801602 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:37.801605 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:37.801711 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 06:45:37.801819 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 06:45:37.801981 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":90,"threads_started":1}
I20260928 06:45:37.802150 13517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.794689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41978 (local address 127.7.48.62:34479)
0928 06:45:37.794827 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:37.794833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:37.795028 (+ 195us) server_negotiation.cc:407] Connection header received
0928 06:45:37.795194 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:37.795197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:37.795251 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:37.795334 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:37.795807 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.796404 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.797253 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.797493 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.797603 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.798774 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:37.798777 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:37.798780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:37.798806 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:37.800495 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.800926 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.800929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.800931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.801022 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.801302 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.801306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.801308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.801503 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:37.801612 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:37.801874 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 06:45:37.801983 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:45:37.802953 13498 tablet_copy_service.cc:140] P 1137a312ab134aa4afef56800ac54199: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3ed6876b7d43478da762015583cd0eeb ({username='oryx', principal='oryx/127.7.48.60@KRBTEST.COM'} at 127.0.0.1:41978)
I20260928 06:45:37.803068 13498 tablet_copy_service.cc:161] P 1137a312ab134aa4afef56800ac54199: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3ed6876b7d43478da762015583cd0eeb at {username='oryx', principal='oryx/127.7.48.60@KRBTEST.COM'} at 127.0.0.1:41978: session id = 3ed6876b7d43478da762015583cd0eeb-00000000000000000000000000000000
I20260928 06:45:37.804013 13498 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 06:45:37.805181 14045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 06:45:37.807137 14045 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb: tablet copy: Starting download of 0 data blocks...
I20260928 06:45:37.807354 14045 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb: tablet copy: Starting download of 1 WAL segments...
I20260928 06:45:37.808785 14045 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 06:45:37.810361 13498 tablet_copy_service.cc:342] P 1137a312ab134aa4afef56800ac54199: Request end of tablet copy session 3ed6876b7d43478da762015583cd0eeb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.7.48.60@KRBTEST.COM'} at 127.0.0.1:41978
I20260928 06:45:37.810510 13498 tablet_copy_service.cc:433] P 1137a312ab134aa4afef56800ac54199: ending tablet copy session 3ed6876b7d43478da762015583cd0eeb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3ed6876b7d43478da762015583cd0eeb
I20260928 06:45:37.817775 14045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:37.819412 14245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:37.819712 14242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:37.820060 14243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:37.820160 14045 server_base.cc:1034] running on GCE node
I20260928 06:45:37.820366 14045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:37.820761 14045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:37.821902 14045 hybrid_clock.cc:648] HybridClock initialized: now 1790577937821887 us; error 42 us; skew 500 ppm
Sep 28 06:45:37 dist-test-slave-twl6 krb5kdc[13433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577937, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:37.823520 14045 init.cc:377] Logged in from keytab as oryx/127.7.48.60@KRBTEST.COM (short username oryx)
I20260928 06:45:37.824075 14045 webserver.cc:460] Webserver started at http://127.7.48.60:42955/ using document root <none> and password file <none>
I20260928 06:45:37.824290 14045 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:37.824347 14045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:37.825475 14045 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:45:37.826431 14252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:37.826663 14045 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:45:37.826821 14045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/wal
uuid: "3ed6876b7d43478da762015583cd0eeb"
format_stamp: "Formatted at 2026-09-28 06:45:37 on dist-test-slave-twl6"
server_key: "4638855de4c45a5a10e2f928f5b857fb"
server_key_iv: "6ea02022373b2ac4e9cc8ca7737c4ad9"
server_key_version: "encryptionkey@0"
I20260928 06:45:37.826987 14045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:37.834162 14045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:37.837831 14045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:37.843041 14045 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.60:40403
I20260928 06:45:37.843581 14045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/master-2/data/info.pb
I20260928 06:45:37.845126 14305 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:45:37.846021 14305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb: Bootstrap starting.
I20260928 06:45:37.852080 14304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.60:40403 every 8 connection(s)
I20260928 06:45:37.853418 14305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:45:37.853749 14305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb: Bootstrap complete.
I20260928 06:45:37.854286 14305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: NON_VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: true } }
I20260928 06:45:37.854421 14305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3ed6876b7d43478da762015583cd0eeb, State: Initialized, Role: LEARNER
I20260928 06:45:37.854545 14305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: NON_VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: true } }
I20260928 06:45:37.854914 14308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: NON_VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: true } } }
I20260928 06:45:37.855096 14305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:45:37.855096 14308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [sys.catalog]: This master's current role is: LEARNER
I20260928 06:45:37.858803 14319 catalog_manager.cc:1259] Loaded cluster ID: 5736c5ca81164407b939a2e06709be0c
I20260928 06:45:37.858922 14319 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb: loading cluster ID for follower catalog manager: success
I20260928 06:45:37.861218 14319 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb: acquiring CA information for follower catalog manager: success
I20260928 06:45:37.862708 14319 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 06:45:37.873320 14224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:37.873636 14224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:37.873744 14224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:37.877897 14224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:37.877993 14224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:37.878103 14224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:45:37.882929 13803 heartbeater.cc:503] Master 127.7.48.62:34479 was elected leader, sending a full tablet report...
I20260928 06:45:37.883806 14224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:44951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.1.keytab
--principal=oryx/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:35937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=39793
--tserver_master_addrs=127.7.48.62:34479,127.7.48.61:33091,127.7.48.60:40403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:37.885114 14224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:37.886161 14224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:37.896337 14325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:37.896324 14324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:37.896324 14327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:37.896669 14224 server_base.cc:1034] running on GCE node
I20260928 06:45:37.897102 14224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:37.897703 14224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:37.898864 14224 hybrid_clock.cc:648] HybridClock initialized: now 1790577937898848 us; error 28 us; skew 500 ppm
Sep 28 06:45:37 dist-test-slave-twl6 krb5kdc[13433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577937, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:37.901727 14224 init.cc:377] Logged in from keytab as oryx/127.7.48.1@KRBTEST.COM (short username oryx)
I20260928 06:45:37.902613 14224 webserver.cc:460] Webserver started at http://127.7.48.1:39793/ using document root <none> and password file <none>
I20260928 06:45:37.903209 14224 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:37.903312 14224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:37.907217 14224 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:37.909318 14334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:37.910576 14224 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:45:37.910717 14224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "99dc5bf901764934b1e36eeeb2d1b276"
format_stamp: "Formatted at 2026-09-28 06:45:36 on dist-test-slave-twl6"
server_key: "f50393b1c9e87b0e7e69581a37adc76d"
server_key_iv: "c130cba8c2975e0d5b201d8d45069e1d"
server_key_version: "encryptionkey@0"
I20260928 06:45:37.911271 14224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:37.939275 14224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:37.942940 14224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:37.943158 14224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:37.943846 14224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:37.944839 14224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:37.944912 14224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:37.944986 14224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:37.945035 14224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:37.954351 14224 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:35937
I20260928 06:45:37.954381 14447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:35937 every 8 connection(s)
I20260928 06:45:37.955538 14224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:45:37.963007 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 14224
I20260928 06:45:37.963186 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 13674
Sep 28 06:45:37 dist-test-slave-twl6 krb5kdc[13433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577937, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.61@KRBTEST.COM
Sep 28 06:45:37 dist-test-slave-twl6 krb5kdc[13433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577937, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM
Sep 28 06:45:37 dist-test-slave-twl6 krb5kdc[13433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577937, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.60@KRBTEST.COM
I20260928 06:45:37.972671 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.957543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:43957 (local address 127.7.48.61:33091)
0928 06:45:37.957723 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:37.957729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:37.958579 (+ 850us) server_negotiation.cc:407] Connection header received
0928 06:45:37.960024 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:37.960028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:37.960082 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:37.960137 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:37.961510 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.962025 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.964230 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.964526 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.964642 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.968172 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:37.968179 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:37.968185 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:37.968219 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:37.969947 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.970649 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.970654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.970659 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.970778 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.971229 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.971235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.971237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.971486 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:37.971569 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:37.972276 (+ 707us) server_negotiation.cc:299] Negotiation successful
0928 06:45:37.972495 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 06:45:37.973912 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.957960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.1:52535)
0928 06:45:37.958427 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:37.958480 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:37.959786 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:37.960291 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:37.960297 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:37.960843 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:37.961361 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.961369 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.962421 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.962425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.963598 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.963613 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.964891 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.964897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.965061 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.965649 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:37.965668 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:37.968736 (+ 3068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:37.971550 (+ 2814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.971552 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.971554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.971804 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.972159 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.972165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.972168 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.972255 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.972718 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:37.972721 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:37.972814 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 06:45:37.972930 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 06:45:37.973077 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":116,"spinlock_wait_cycles":39552,"thread_start_us":150,"threads_started":1}
I20260928 06:45:37.974326 13517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.957565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:52535 (local address 127.7.48.62:34479)
0928 06:45:37.957703 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:37.957708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:37.958587 (+ 879us) server_negotiation.cc:407] Connection header received
0928 06:45:37.960014 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:37.960018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:37.960078 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:37.960157 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:37.961509 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.962295 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.964199 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.964616 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.964737 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.968903 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:37.968908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:37.968910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:37.968943 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:37.971414 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.971942 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.971945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.971947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.972046 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.972361 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.972365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.972366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.972578 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:37.972669 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:37.974038 (+ 1369us) server_negotiation.cc:299] Negotiation successful
0928 06:45:37.974176 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 06:45:37.974855 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.957960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:33091 (local address 127.7.48.1:43957)
0928 06:45:37.958427 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:37.958484 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:37.959768 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:37.960294 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:37.960302 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:37.960833 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:37.961346 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.961360 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.962168 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.962171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.964051 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.964111 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.964674 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.964678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.964872 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.965564 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:37.965590 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:37.967978 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:37.970108 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.970116 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.970135 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.970512 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.970909 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.970912 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.970915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.971104 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.971634 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:37.971640 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:37.971981 (+ 341us) client_negotiation.cc:769] Sending connection context
0928 06:45:37.973109 (+ 1128us) client_negotiation.cc:240] Negotiation successful
0928 06:45:37.973228 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":37,"spinlock_wait_cycles":6144,"thread_start_us":134,"threads_started":1}
I20260928 06:45:37.975062 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:43311
--local_ip_for_outbound_sockets=127.7.48.2
--tserver_master_addrs=127.7.48.62:34479
--webserver_port=36247
--webserver_interface=127.7.48.2
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.2.keytab
--principal=oryx/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:44951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.48.62:34479,127.7.48.61:33091
--tserver_master_addrs=127.7.48.62:34479,127.7.48.61:33091,127.7.48.60:40403 with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:45:37.975234 14455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.959095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:37413 (local address 127.7.48.60:40403)
0928 06:45:37.959329 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:37.959343 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:37.959425 (+ 82us) server_negotiation.cc:407] Connection header received
0928 06:45:37.960274 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:37.960278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:37.960329 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:37.960417 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:37.961514 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.962549 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.965562 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.966244 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.966363 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.968914 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:37.968921 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:37.968927 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:37.968959 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:37.970751 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.971290 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.971296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.971301 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.971391 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:37.971690 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:37.971697 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:37.971701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:37.971992 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:37.972093 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:37.974921 (+ 2828us) server_negotiation.cc:299] Negotiation successful
0928 06:45:37.975078 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":79,"threads_started":1}
I20260928 06:45:37.973914 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:37.958964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:40403 (local address 127.7.48.1:37413)
0928 06:45:37.959346 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:37.959357 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:37.959786 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:37.960449 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:37.960454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:37.960843 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:37.961359 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.961369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.962691 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.962694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.965401 (+ 2707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:37.965414 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:37.966517 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:37.966523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:37.966652 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:37.967145 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:37.967164 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:37.968754 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:37.970881 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.970887 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.970892 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.971174 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.971490 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:37.971493 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:37.971495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:37.971588 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:37.972144 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:37.972149 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:37.972276 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 06:45:37.972401 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 06:45:37.972773 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":1965,"spinlock_wait_cycles":13312,"thread_start_us":83,"threads_started":1}
I20260928 06:45:37.975917 14450 heartbeater.cc:347] Connected to a master server at 127.7.48.61:33091
I20260928 06:45:37.976182 14450 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:37.976686 14450 heartbeater.cc:511] Master 127.7.48.61:33091 requested a full tablet report, sending...
I20260928 06:45:37.976981 14448 heartbeater.cc:347] Connected to a master server at 127.7.48.60:40403
I20260928 06:45:37.976987 14449 heartbeater.cc:347] Connected to a master server at 127.7.48.62:34479
I20260928 06:45:37.977069 14448 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:37.977108 14449 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:37.977239 14448 heartbeater.cc:511] Master 127.7.48.60:40403 requested a full tablet report, sending...
I20260928 06:45:37.977334 14449 heartbeater.cc:511] Master 127.7.48.62:34479 requested a full tablet report, sending...
I20260928 06:45:37.978235 13477 ts_manager.cc:194] Re-registered known tserver with Master: 99dc5bf901764934b1e36eeeb2d1b276 (127.7.48.1:35937)
I20260928 06:45:37.978652 14268 ts_manager.cc:194] Registered new tserver with Master: 99dc5bf901764934b1e36eeeb2d1b276 (127.7.48.1:35937)
I20260928 06:45:37.978904 13477 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.48.1@KRBTEST.COM'} at 127.7.48.1:52535
I20260928 06:45:37.979830 14080 ts_manager.cc:194] Registered new tserver with Master: 99dc5bf901764934b1e36eeeb2d1b276 (127.7.48.1:35937)
W20260928 06:45:38.088080 14456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:38.088428 14456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:38.088507 14456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:38.092727 14456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:38.092795 14456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:38.092887 14456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:45:38.098263 14456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:44951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.2.keytab
--principal=oryx/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:43311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=36247
--tserver_master_addrs=127.7.48.62:34479,127.7.48.61:33091,127.7.48.60:40403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:38.099222 14456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:38.100226 14456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:38.109902 14464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:38.110008 14456 server_base.cc:1034] running on GCE node
W20260928 06:45:38.109961 14462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:38.109918 14461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:38.110687 14456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:38.110772 13937 heartbeater.cc:503] Master 127.7.48.62:34479 was elected leader, sending a full tablet report...
I20260928 06:45:38.111478 14456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:38.112679 14456 hybrid_clock.cc:648] HybridClock initialized: now 1790577938112656 us; error 41 us; skew 500 ppm
Sep 28 06:45:38 dist-test-slave-twl6 krb5kdc[13433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577938, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:38.115587 14456 init.cc:377] Logged in from keytab as oryx/127.7.48.2@KRBTEST.COM (short username oryx)
I20260928 06:45:38.116528 14456 webserver.cc:460] Webserver started at http://127.7.48.2:36247/ using document root <none> and password file <none>
I20260928 06:45:38.117153 14456 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:38.117270 14456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:38.121635 14456 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:45:38.123723 14471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:38.125007 14456 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:45:38.125149 14456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "3b0b1b727b634600a687052b21a952b2"
format_stamp: "Formatted at 2026-09-28 06:45:36 on dist-test-slave-twl6"
server_key: "298ec7ab345119ffd429a16183c132ea"
server_key_iv: "18cb796e7ad6f9f853fdbe901e65d459"
server_key_version: "encryptionkey@0"
I20260928 06:45:38.125715 14456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:38.136832 14456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:38.139843 14456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:38.140084 14456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:38.140677 14456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:38.141539 14456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:38.141609 14456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:38.141683 14456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:38.141736 14456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:38.150591 14456 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:43311
I20260928 06:45:38.150624 14584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:43311 every 8 connection(s)
I20260928 06:45:38.151644 14456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:45:38.152084 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 14456
I20260928 06:45:38.152279 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 13808
I20260928 06:45:38.160432 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:41911
--local_ip_for_outbound_sockets=127.7.48.3
--tserver_master_addrs=127.7.48.62:34479
--webserver_port=41257
--webserver_interface=127.7.48.3
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.3.keytab
--principal=oryx/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:44951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.48.62:34479,127.7.48.61:33091
--tserver_master_addrs=127.7.48.62:34479,127.7.48.61:33091,127.7.48.60:40403 with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:45:38 dist-test-slave-twl6 krb5kdc[13433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577938, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM
Sep 28 06:45:38 dist-test-slave-twl6 krb5kdc[13433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577938, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.60@KRBTEST.COM
Sep 28 06:45:38 dist-test-slave-twl6 krb5kdc[13433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577938, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.61@KRBTEST.COM
I20260928 06:45:38.173162 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.155683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:49437 (local address 127.7.48.61:33091)
0928 06:45:38.155832 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:38.155839 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:38.156268 (+ 429us) server_negotiation.cc:407] Connection header received
0928 06:45:38.157423 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:38.157427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:38.157485 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:38.157543 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:38.159731 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.160605 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.162683 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.163033 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.163110 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.167968 (+ 4858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:38.167972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:38.167974 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:38.168006 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:38.170263 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.171052 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.171055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.171057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.171267 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.171673 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.171677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.171679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.171947 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:38.172013 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:38.172842 (+ 829us) server_negotiation.cc:299] Negotiation successful
0928 06:45:38.172986 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:45:38.173221 13517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.153888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:44765 (local address 127.7.48.62:34479)
0928 06:45:38.154022 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:38.154026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:38.156086 (+ 2060us) server_negotiation.cc:407] Connection header received
0928 06:45:38.157433 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:38.157440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:38.157509 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:38.157570 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:38.159732 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.160568 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.162692 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.163066 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.163151 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.168005 (+ 4854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:38.168011 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:38.168017 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:38.168055 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:38.170194 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.171088 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.171096 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.171100 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.171205 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.171707 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.171713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.171716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.172013 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:38.172103 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:38.172853 (+ 750us) server_negotiation.cc:299] Negotiation successful
0928 06:45:38.173000 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 06:45:38.173285 14455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.155839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:57045 (local address 127.7.48.60:40403)
0928 06:45:38.155989 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:38.155995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:38.156013 (+ 18us) server_negotiation.cc:407] Connection header received
0928 06:45:38.157424 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:38.157427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:38.157502 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:38.157554 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:38.159732 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.160597 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.162683 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.163033 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.163115 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.167975 (+ 4860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:38.167979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:38.167981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:38.168013 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:38.170280 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.171052 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.171055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.171057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.171166 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.171665 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.171668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.171670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.171927 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:38.172004 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:38.172991 (+ 987us) server_negotiation.cc:299] Negotiation successful
0928 06:45:38.173120 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 06:45:38.174948 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.154069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.2:44765)
0928 06:45:38.155981 (+ 1912us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:38.155997 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:38.157159 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:38.157965 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:38.157975 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:38.158628 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:38.159546 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:38.159562 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.160970 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.160976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:38.162516 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:38.162529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.163343 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.163347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:38.163654 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.164616 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:38.164639 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:38.167776 (+ 3137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:38.170361 (+ 2585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:38.170370 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:38.170392 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:38.170952 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:38.171341 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:38.171347 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:38.171350 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:38.171567 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:38.172194 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:38.172199 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:38.172514 (+ 315us) client_negotiation.cc:769] Sending connection context
0928 06:45:38.173405 (+ 891us) client_negotiation.cc:240] Negotiation successful
0928 06:45:38.173808 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1802,"mutex_wait_us":485,"spinlock_wait_cycles":56960,"thread_start_us":168,"threads_started":1}
I20260928 06:45:38.174952 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.155229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:33091 (local address 127.7.48.2:49437)
0928 06:45:38.156184 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:38.156197 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:38.157165 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:38.157965 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:38.157974 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:38.158647 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:38.159542 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:38.159562 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.160787 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.160790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:38.162541 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:38.162552 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.163322 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.163325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:38.163647 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.164528 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:38.164616 (+ 88us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:38.167731 (+ 3115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:38.170410 (+ 2679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:38.170414 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:38.170416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:38.170945 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:38.171367 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:38.171374 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:38.171376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:38.171552 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:38.172092 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:38.172099 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:38.172515 (+ 416us) client_negotiation.cc:769] Sending connection context
0928 06:45:38.173928 (+ 1413us) client_negotiation.cc:240] Negotiation successful
0928 06:45:38.174043 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":647,"spinlock_wait_cycles":146176,"thread_start_us":105,"threads_started":1}
I20260928 06:45:38.174965 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.154069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:40403 (local address 127.7.48.2:57045)
0928 06:45:38.155626 (+ 1557us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:38.155684 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:38.157165 (+ 1481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:38.157965 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:38.157974 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:38.158647 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:38.159542 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:38.159563 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.160787 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.160790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:38.162542 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:38.162552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.163321 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.163325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:38.163652 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.164740 (+ 1088us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:38.164756 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:38.167731 (+ 2975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:38.170410 (+ 2679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:38.170414 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:38.170416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:38.170928 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:38.171409 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:38.171414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:38.171419 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:38.171547 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:38.172092 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:38.172099 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:38.172511 (+ 412us) client_negotiation.cc:769] Sending connection context
0928 06:45:38.173593 (+ 1082us) client_negotiation.cc:240] Negotiation successful
0928 06:45:38.173898 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1308,"mutex_wait_us":712,"spinlock_wait_cycles":24576,"thread_start_us":129,"threads_started":1}
I20260928 06:45:38.176163 14586 heartbeater.cc:347] Connected to a master server at 127.7.48.62:34479
I20260928 06:45:38.176234 14585 heartbeater.cc:347] Connected to a master server at 127.7.48.60:40403
I20260928 06:45:38.176403 14585 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:38.176403 14586 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:38.176427 14587 heartbeater.cc:347] Connected to a master server at 127.7.48.61:33091
I20260928 06:45:38.176548 14587 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:38.177017 14586 heartbeater.cc:511] Master 127.7.48.62:34479 requested a full tablet report, sending...
I20260928 06:45:38.177018 14587 heartbeater.cc:511] Master 127.7.48.61:33091 requested a full tablet report, sending...
I20260928 06:45:38.177018 14585 heartbeater.cc:511] Master 127.7.48.60:40403 requested a full tablet report, sending...
I20260928 06:45:38.178190 13477 ts_manager.cc:194] Re-registered known tserver with Master: 3b0b1b727b634600a687052b21a952b2 (127.7.48.2:43311)
I20260928 06:45:38.178313 14268 ts_manager.cc:194] Registered new tserver with Master: 3b0b1b727b634600a687052b21a952b2 (127.7.48.2:43311)
I20260928 06:45:38.178323 14080 ts_manager.cc:194] Registered new tserver with Master: 3b0b1b727b634600a687052b21a952b2 (127.7.48.2:43311)
I20260928 06:45:38.178825 13477 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.48.2@KRBTEST.COM'} at 127.7.48.2:44765
I20260928 06:45:38.259863 14455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.256467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59554 (local address 127.7.48.60:40403)
0928 06:45:38.256606 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:38.256613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:38.256662 (+ 49us) server_negotiation.cc:407] Connection header received
0928 06:45:38.256858 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:38.256864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:38.256960 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:38.257072 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:38.257337 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.257860 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.258885 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.259326 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.259446 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.259610 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:38.259648 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 06:45:38.259716 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 06:45:38.259861 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.256365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:40403 (local address 127.0.0.1:59554)
0928 06:45:38.256569 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:38.256586 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:38.256732 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:38.257085 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:38.257089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:38.257107 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:38.257220 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:38.257226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.257972 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.257977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:38.258751 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:38.258760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.259451 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.259455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:38.259537 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.259560 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 06:45:38.259626 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:45:38.259719 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 06:45:38.260576 14279 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 06:45:38.260864 14308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1137a312ab134aa4afef56800ac54199. Latest consensus state: current_term: 1 leader_uuid: "1137a312ab134aa4afef56800ac54199" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: NON_VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: true } } }
I20260928 06:45:38.261037 14308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [sys.catalog]: This master's current role is: LEARNER
W20260928 06:45:38.285140 14592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:38.285440 14592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:38.285511 14592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:38.289973 14592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:38.290061 14592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:38.290169 14592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:45:38.295534 14592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:44951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.3.keytab
--principal=oryx/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:41911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=41257
--tserver_master_addrs=127.7.48.62:34479,127.7.48.61:33091,127.7.48.60:40403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:38.296741 14592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:38.297853 14592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:38.307703 14600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:38.307700 14598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:38.307701 14597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:38.307979 14592 server_base.cc:1034] running on GCE node
I20260928 06:45:38.308454 14592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:38.309089 14592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:38.310292 14592 hybrid_clock.cc:648] HybridClock initialized: now 1790577938310247 us; error 53 us; skew 500 ppm
Sep 28 06:45:38 dist-test-slave-twl6 krb5kdc[13433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577938, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:38.312984 14592 init.cc:377] Logged in from keytab as oryx/127.7.48.3@KRBTEST.COM (short username oryx)
I20260928 06:45:38.313885 14592 webserver.cc:460] Webserver started at http://127.7.48.3:41257/ using document root <none> and password file <none>
I20260928 06:45:38.314525 14592 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:38.314639 14592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:38.318755 14592 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:45:38.321149 14607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:38.322397 14592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:45:38.322528 14592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6"
format_stamp: "Formatted at 2026-09-28 06:45:37 on dist-test-slave-twl6"
server_key: "8ea3148451a07e48626a6f4b6327813c"
server_key_iv: "184ccfc019646f9a4a27aae6dbac4d4f"
server_key_version: "encryptionkey@0"
I20260928 06:45:38.323083 14592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:38.333166 14592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:38.335827 14592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:38.336050 14592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:38.336675 14592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:38.337544 14592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:38.337615 14592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:38.337674 14592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:38.337719 14592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:38.348213 14592 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:41911
I20260928 06:45:38.348232 14720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:41911 every 8 connection(s)
I20260928 06:45:38.349246 14592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:45:38.358073 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 14592
Sep 28 06:45:38 dist-test-slave-twl6 krb5kdc[13433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577938, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.60@KRBTEST.COM
Sep 28 06:45:38 dist-test-slave-twl6 krb5kdc[13433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577938, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.61@KRBTEST.COM
Sep 28 06:45:38 dist-test-slave-twl6 krb5kdc[13433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577938, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM
I20260928 06:45:38.368232 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.353086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:45757 (local address 127.7.48.61:33091)
0928 06:45:38.353211 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:38.353216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:38.353378 (+ 162us) server_negotiation.cc:407] Connection header received
0928 06:45:38.353839 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:38.353842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:38.353900 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:38.353979 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:38.355462 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.356266 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.357794 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.358114 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.358211 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.362654 (+ 4443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:38.362659 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:38.362665 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:38.362699 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:38.364707 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.366327 (+ 1620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.366332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.366334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.366423 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.366841 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.366846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.366851 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.367072 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:38.367155 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:38.367915 (+ 760us) server_negotiation.cc:299] Negotiation successful
0928 06:45:38.368035 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:45:38.368458 13517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.351062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:44141 (local address 127.7.48.62:34479)
0928 06:45:38.351186 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:38.351190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:38.352613 (+ 1423us) server_negotiation.cc:407] Connection header received
0928 06:45:38.353865 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:38.353869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:38.353925 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:38.353980 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:38.355476 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.356356 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.360458 (+ 4102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.360822 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.360917 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.364721 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:38.364726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:38.364732 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:38.364766 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:38.366667 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.367124 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.367128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.367129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.367217 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.367502 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.367505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.367506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.367715 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:38.367811 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:38.368170 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 06:45:38.368301 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:45:38.369933 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.352988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:33091 (local address 127.7.48.3:45757)
0928 06:45:38.353286 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:38.353298 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:38.353630 (+ 332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:38.354158 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:38.354165 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:38.354660 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:38.355315 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:38.355330 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.356411 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.356417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:38.357676 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:38.357686 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.358391 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.358395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:38.358737 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.359530 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:38.359575 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:38.362479 (+ 2904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:38.365779 (+ 3300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:38.365788 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:38.365807 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:38.366206 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:38.366543 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:38.366546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:38.366548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:38.366731 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:38.367241 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:38.367251 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:38.367595 (+ 344us) client_negotiation.cc:769] Sending connection context
0928 06:45:38.368381 (+ 786us) client_negotiation.cc:240] Negotiation successful
0928 06:45:38.368779 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":216,"spinlock_wait_cycles":222336,"thread_start_us":101,"threads_started":1}
I20260928 06:45:38.370944 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.360416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41988 (local address 127.7.48.62:34479)
0928 06:45:38.360668 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:38.360674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:38.360689 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:45:38.360726 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:38.360729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:38.360779 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:38.361881 (+ 1102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:38.361989 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.362794 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.364171 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.364575 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.365068 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.366489 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:38.366495 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:38.366498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:38.366521 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:38.368820 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.369302 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.369305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.369307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.369410 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.369727 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.369730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.369732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.369947 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:38.370567 (+ 620us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:38.370651 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 06:45:38.370765 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":79,"threads_started":1}
I20260928 06:45:38.371445 14723 heartbeater.cc:347] Connected to a master server at 127.7.48.61:33091
I20260928 06:45:38.371706 14723 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:38.372037 14455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.351698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:45723 (local address 127.7.48.60:40403)
0928 06:45:38.351832 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:38.351836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:38.352601 (+ 765us) server_negotiation.cc:407] Connection header received
0928 06:45:38.353866 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:38.353869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:38.353930 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:38.353980 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:38.355476 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.356318 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.357857 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.358203 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.358293 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.362789 (+ 4496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:38.362793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:38.362796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:38.362825 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:38.365613 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.370646 (+ 5033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.370649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.370650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.370740 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.371077 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.371080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.371082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.371302 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:38.371384 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:38.371739 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 06:45:38.371857 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 06:45:38.372310 14723 heartbeater.cc:511] Master 127.7.48.61:33091 requested a full tablet report, sending...
I20260928 06:45:38.372491 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.351848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:40403 (local address 127.7.48.3:45723)
0928 06:45:38.352429 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:38.352492 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:38.353625 (+ 1133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:38.354159 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:38.354167 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:38.354662 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:38.355328 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:38.355337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.356470 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.356474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:38.357709 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:38.357720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.358398 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.358401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:38.358741 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.359612 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:38.359627 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:38.362640 (+ 3013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:38.370303 (+ 7663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:38.370306 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:38.370308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:38.370538 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:38.370874 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:38.370880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:38.370885 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:38.370973 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:38.371445 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:38.371447 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:38.371548 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 06:45:38.372170 (+ 622us) client_negotiation.cc:240] Negotiation successful
0928 06:45:38.372307 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":382,"spinlock_wait_cycles":700288,"thread_start_us":126,"threads_started":1}
I20260928 06:45:38.374009 14721 heartbeater.cc:347] Connected to a master server at 127.7.48.60:40403
I20260928 06:45:38.374085 14721 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:38.374153 14080 ts_manager.cc:194] Registered new tserver with Master: f84f119cbe8f49f5b7d2c0fd0009a3b6 (127.7.48.3:41911)
I20260928 06:45:38.374240 14721 heartbeater.cc:511] Master 127.7.48.60:40403 requested a full tablet report, sending...
I20260928 06:45:38.375025 14268 ts_manager.cc:194] Registered new tserver with Master: f84f119cbe8f49f5b7d2c0fd0009a3b6 (127.7.48.3:41911)
I20260928 06:45:38.375196 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.351848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.3:44141)
0928 06:45:38.352429 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:38.352492 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:38.353625 (+ 1133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:38.354159 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:38.354167 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:38.354682 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:38.355329 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:38.355337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.356482 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.356484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:38.357709 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:38.357720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.361198 (+ 3478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.361202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:38.361314 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.362436 (+ 1122us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:38.362450 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:38.364539 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:38.366780 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:38.366784 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:38.366786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:38.367010 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:38.367317 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:38.367320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:38.367321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:38.367401 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:38.367867 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:38.367869 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:38.367976 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 06:45:38.368675 (+ 699us) client_negotiation.cc:240] Negotiation successful
0928 06:45:38.368827 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":387,"spinlock_wait_cycles":87424,"thread_start_us":172,"threads_started":1}
I20260928 06:45:38.376032 14722 heartbeater.cc:347] Connected to a master server at 127.7.48.62:34479
I20260928 06:45:38.376113 14722 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:38.376281 14722 heartbeater.cc:511] Master 127.7.48.62:34479 requested a full tablet report, sending...
I20260928 06:45:38.376899 13478 ts_manager.cc:194] Re-registered known tserver with Master: f84f119cbe8f49f5b7d2c0fd0009a3b6 (127.7.48.3:41911)
I20260928 06:45:38.377709 13478 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.48.3@KRBTEST.COM'} at 127.7.48.3:44141
I20260928 06:45:38.374307 13477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:45:38.379362 13477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:45:38.379715 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.360820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49406 (local address 127.7.48.61:33091)
0928 06:45:38.362884 (+ 2064us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:38.362890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:38.362903 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:45:38.363362 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:38.363383 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:38.363436 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:38.363528 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:38.365608 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.368794 (+ 3186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.370567 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.370910 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.371016 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.374187 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:38.374190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:38.374193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:38.374216 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:38.377410 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.377872 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.377876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.377877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.377967 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.378413 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.378416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.378418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.378663 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:38.378788 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:38.379423 (+ 635us) server_negotiation.cc:299] Negotiation successful
0928 06:45:38.379560 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1986,"thread_start_us":92,"threads_started":1}
I20260928 06:45:38.381424 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.361921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59570 (local address 127.7.48.60:40403)
0928 06:45:38.362442 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:38.362446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:38.365470 (+ 3024us) server_negotiation.cc:407] Connection header received
0928 06:45:38.366518 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:38.366521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:38.366574 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:38.366658 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:38.367553 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.368414 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.371985 (+ 3571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.372751 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.372868 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.376584 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:38.376590 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:38.376593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:38.376618 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:38.378360 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.380243 (+ 1883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.380246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.380248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.380341 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.380671 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.380674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.380675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.380845 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:38.380939 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:38.381179 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 06:45:38.381271 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":102,"threads_started":1}
I20260928 06:45:38.382412 14740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1137a312ab134aa4afef56800ac54199" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: NON_VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: true } } }
I20260928 06:45:38.382539 14740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:38.384349 14039 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199: attempting to promote NON_VOTER 3ed6876b7d43478da762015583cd0eeb to VOTER
I20260928 06:45:38.384848 14039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: false } }
I20260928 06:45:38.386086 14090 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 1 FOLLOWER]: Refusing update from remote peer 1137a312ab134aa4afef56800ac54199: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 06:45:38.386188 14279 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 1 LEARNER]: Refusing update from remote peer 1137a312ab134aa4afef56800ac54199: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 06:45:38.386445 14740 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 06:45:38.386579 14039 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 06:45:38.388499 14739 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3ed6876b7d43478da762015583cd0eeb (127.7.48.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: false } } }
I20260928 06:45:38.388839 14279 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3ed6876b7d43478da762015583cd0eeb (127.7.48.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: false } } }
I20260928 06:45:38.388931 14090 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3ed6876b7d43478da762015583cd0eeb (127.7.48.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: false } } }
I20260928 06:45:38.390276 14308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1137a312ab134aa4afef56800ac54199" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: false } } }
I20260928 06:45:38.390395 14308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:45:38.390705 14748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1137a312ab134aa4afef56800ac54199" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: false } } }
I20260928 06:45:38.390844 14748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:45:38.391248 14039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1137a312ab134aa4afef56800ac54199" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: false } } }
I20260928 06:45:38.391361 14039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:38.392033 14739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1137a312ab134aa4afef56800ac54199" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: false } } }
I20260928 06:45:38.392159 14739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:38.392112 14749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1137a312ab134aa4afef56800ac54199" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: false } } }
I20260928 06:45:38.392277 14749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:38.403806 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.397427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:35937 (local address 127.0.0.1:40342)
0928 06:45:38.397576 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:38.397590 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:38.397666 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:38.399455 (+ 1789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:38.399458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:38.399476 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:38.399596 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:38.399606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.401022 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.401025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:38.402327 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:38.402338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.403280 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.403283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:38.403418 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.403492 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 06:45:38.403558 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:45:38.403641 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":212}
I20260928 06:45:38.403849 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.398308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36376 (local address 127.7.48.2:43311)
0928 06:45:38.399096 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:38.399108 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:38.399130 (+ 22us) server_negotiation.cc:407] Connection header received
0928 06:45:38.399201 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:38.399206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:38.399356 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:38.399804 (+ 448us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:38.399966 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.401019 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.402265 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.403105 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.403234 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.403394 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:38.403589 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 06:45:38.403700 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":103,"threads_started":1}
I20260928 06:45:38.403808 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.397738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:43311 (local address 127.0.0.1:36376)
0928 06:45:38.398006 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:38.398021 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:38.398108 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:38.399540 (+ 1432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:38.399544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:38.399561 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:38.399649 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:38.399656 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.401107 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.401112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:38.402119 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:38.402130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.403265 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.403268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:38.403414 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.403461 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 06:45:38.403544 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 06:45:38.403641 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":38,"thread_start_us":112,"threads_started":1}
I20260928 06:45:38.404539 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.397871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40342 (local address 127.7.48.1:35937)
0928 06:45:38.398248 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:38.398700 (+ 452us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:38.398744 (+ 44us) server_negotiation.cc:407] Connection header received
0928 06:45:38.398849 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:38.398854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:38.399280 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:38.399460 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:38.399720 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.400907 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.402447 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.403180 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.403276 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.403393 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:38.404124 (+ 731us) server_negotiation.cc:299] Negotiation successful
0928 06:45:38.404369 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":111,"threads_started":1}
I20260928 06:45:38.405130 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.398125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:41911 (local address 127.0.0.1:43188)
0928 06:45:38.398408 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:38.398424 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:38.398509 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:38.399811 (+ 1302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:38.399813 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:38.399826 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:38.399934 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:38.399940 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.400889 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.400894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:38.402327 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:38.402338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.404690 (+ 2352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.404694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:38.404788 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.404824 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 06:45:38.404924 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 06:45:38.404994 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":78,"thread_start_us":120,"threads_started":1}
I20260928 06:45:38.407217 14519 tablet_service.cc:1467] Processing CreateTablet for tablet 45a75e4f057e44d8b20e510bcd141305 (DEFAULT_TABLE table=test-table [id=2126191723384503a39bec35700bd6e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:38.407219 14382 tablet_service.cc:1467] Processing CreateTablet for tablet 45a75e4f057e44d8b20e510bcd141305 (DEFAULT_TABLE table=test-table [id=2126191723384503a39bec35700bd6e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:38.408128 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.398646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43188 (local address 127.7.48.3:41911)
0928 06:45:38.398995 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:38.399006 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:38.399026 (+ 20us) server_negotiation.cc:407] Connection header received
0928 06:45:38.399100 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:38.399105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:38.399502 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:38.399673 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:38.400047 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.400774 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.403468 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.404334 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.404469 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.404601 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:38.404969 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 06:45:38.405065 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":131,"threads_started":1}
I20260928 06:45:38.408274 14519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45a75e4f057e44d8b20e510bcd141305. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:38.408274 14382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45a75e4f057e44d8b20e510bcd141305. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:38.410946 14655 tablet_service.cc:1467] Processing CreateTablet for tablet 45a75e4f057e44d8b20e510bcd141305 (DEFAULT_TABLE table=test-table [id=2126191723384503a39bec35700bd6e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:38.412003 14655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45a75e4f057e44d8b20e510bcd141305. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:38.415362 14755 tablet_bootstrap.cc:492] T 45a75e4f057e44d8b20e510bcd141305 P 3b0b1b727b634600a687052b21a952b2: Bootstrap starting.
I20260928 06:45:38.417675 14755 tablet_bootstrap.cc:654] T 45a75e4f057e44d8b20e510bcd141305 P 3b0b1b727b634600a687052b21a952b2: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:38.418396 14755 log.cc:824] T 45a75e4f057e44d8b20e510bcd141305 P 3b0b1b727b634600a687052b21a952b2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:38.419018 14756 tablet_bootstrap.cc:492] T 45a75e4f057e44d8b20e510bcd141305 P 99dc5bf901764934b1e36eeeb2d1b276: Bootstrap starting.
I20260928 06:45:38.419837 14758 tablet_bootstrap.cc:492] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6: Bootstrap starting.
I20260928 06:45:38.420645 14755 tablet_bootstrap.cc:492] T 45a75e4f057e44d8b20e510bcd141305 P 3b0b1b727b634600a687052b21a952b2: No bootstrap required, opened a new log
I20260928 06:45:38.420850 14755 ts_tablet_manager.cc:1397] T 45a75e4f057e44d8b20e510bcd141305 P 3b0b1b727b634600a687052b21a952b2: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260928 06:45:38.421962 14756 tablet_bootstrap.cc:654] T 45a75e4f057e44d8b20e510bcd141305 P 99dc5bf901764934b1e36eeeb2d1b276: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:38.422012 14758 tablet_bootstrap.cc:654] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:38.422848 14756 log.cc:824] T 45a75e4f057e44d8b20e510bcd141305 P 99dc5bf901764934b1e36eeeb2d1b276: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:38.422848 14758 log.cc:824] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:38.424947 14755 raft_consensus.cc:348] T 45a75e4f057e44d8b20e510bcd141305 P 3b0b1b727b634600a687052b21a952b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99dc5bf901764934b1e36eeeb2d1b276" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35937 } } peers { permanent_uuid: "3b0b1b727b634600a687052b21a952b2" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 43311 } } peers { permanent_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41911 } }
I20260928 06:45:38.425191 14755 raft_consensus.cc:374] T 45a75e4f057e44d8b20e510bcd141305 P 3b0b1b727b634600a687052b21a952b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:38.425254 14755 raft_consensus.cc:729] T 45a75e4f057e44d8b20e510bcd141305 P 3b0b1b727b634600a687052b21a952b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b0b1b727b634600a687052b21a952b2, State: Initialized, Role: FOLLOWER
I20260928 06:45:38.425869 14758 tablet_bootstrap.cc:492] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6: No bootstrap required, opened a new log
I20260928 06:45:38.425869 14756 tablet_bootstrap.cc:492] T 45a75e4f057e44d8b20e510bcd141305 P 99dc5bf901764934b1e36eeeb2d1b276: No bootstrap required, opened a new log
I20260928 06:45:38.425909 14755 consensus_queue.cc:260] T 45a75e4f057e44d8b20e510bcd141305 P 3b0b1b727b634600a687052b21a952b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99dc5bf901764934b1e36eeeb2d1b276" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35937 } } peers { permanent_uuid: "3b0b1b727b634600a687052b21a952b2" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 43311 } } peers { permanent_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41911 } }
I20260928 06:45:38.426070 14756 ts_tablet_manager.cc:1397] T 45a75e4f057e44d8b20e510bcd141305 P 99dc5bf901764934b1e36eeeb2d1b276: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 06:45:38.426070 14758 ts_tablet_manager.cc:1397] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260928 06:45:38.427052 14755 ts_tablet_manager.cc:1428] T 45a75e4f057e44d8b20e510bcd141305 P 3b0b1b727b634600a687052b21a952b2: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260928 06:45:38.427412 14586 heartbeater.cc:503] Master 127.7.48.62:34479 was elected leader, sending a full tablet report...
I20260928 06:45:38.430902 14758 raft_consensus.cc:348] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99dc5bf901764934b1e36eeeb2d1b276" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35937 } } peers { permanent_uuid: "3b0b1b727b634600a687052b21a952b2" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 43311 } } peers { permanent_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41911 } }
I20260928 06:45:38.431176 14758 raft_consensus.cc:374] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:38.431237 14758 raft_consensus.cc:729] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f84f119cbe8f49f5b7d2c0fd0009a3b6, State: Initialized, Role: FOLLOWER
I20260928 06:45:38.431705 14758 consensus_queue.cc:260] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99dc5bf901764934b1e36eeeb2d1b276" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35937 } } peers { permanent_uuid: "3b0b1b727b634600a687052b21a952b2" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 43311 } } peers { permanent_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41911 } }
I20260928 06:45:38.432435 14722 heartbeater.cc:503] Master 127.7.48.62:34479 was elected leader, sending a full tablet report...
I20260928 06:45:38.432822 14758 ts_tablet_manager.cc:1428] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260928 06:45:38.434422 14756 raft_consensus.cc:348] T 45a75e4f057e44d8b20e510bcd141305 P 99dc5bf901764934b1e36eeeb2d1b276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99dc5bf901764934b1e36eeeb2d1b276" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35937 } } peers { permanent_uuid: "3b0b1b727b634600a687052b21a952b2" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 43311 } } peers { permanent_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41911 } }
I20260928 06:45:38.434705 14756 raft_consensus.cc:374] T 45a75e4f057e44d8b20e510bcd141305 P 99dc5bf901764934b1e36eeeb2d1b276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:38.434772 14756 raft_consensus.cc:729] T 45a75e4f057e44d8b20e510bcd141305 P 99dc5bf901764934b1e36eeeb2d1b276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99dc5bf901764934b1e36eeeb2d1b276, State: Initialized, Role: FOLLOWER
I20260928 06:45:38.435273 14756 consensus_queue.cc:260] T 45a75e4f057e44d8b20e510bcd141305 P 99dc5bf901764934b1e36eeeb2d1b276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99dc5bf901764934b1e36eeeb2d1b276" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35937 } } peers { permanent_uuid: "3b0b1b727b634600a687052b21a952b2" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 43311 } } peers { permanent_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41911 } }
I20260928 06:45:38.436110 14449 heartbeater.cc:503] Master 127.7.48.62:34479 was elected leader, sending a full tablet report...
I20260928 06:45:38.436421 14756 ts_tablet_manager.cc:1428] T 45a75e4f057e44d8b20e510bcd141305 P 99dc5bf901764934b1e36eeeb2d1b276: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
W20260928 06:45:38.457720 14451 tablet.cc:2369] T 45a75e4f057e44d8b20e510bcd141305 P 99dc5bf901764934b1e36eeeb2d1b276: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:45:38.465163 14762 raft_consensus.cc:482] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:45:38.465371 14762 raft_consensus.cc:504] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99dc5bf901764934b1e36eeeb2d1b276" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35937 } } peers { permanent_uuid: "3b0b1b727b634600a687052b21a952b2" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 43311 } } peers { permanent_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41911 } }
I20260928 06:45:38.466552 14762 leader_election.cc:288] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99dc5bf901764934b1e36eeeb2d1b276 (127.7.48.1:35937), 3b0b1b727b634600a687052b21a952b2 (127.7.48.2:43311)
I20260928 06:45:38.470189 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.466819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:35937 (local address 127.7.48.3:57509)
0928 06:45:38.466929 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:38.466946 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:38.467044 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:38.467332 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:38.467334 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:38.467354 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:38.467497 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:38.467503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.468280 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.468285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:38.469106 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:38.469114 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.469753 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.469757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:38.469866 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.469885 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 06:45:38.469973 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:45:38.470046 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":65,"spinlock_wait_cycles":13440}
I20260928 06:45:38.470188 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.466939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:57509 (local address 127.7.48.1:35937)
0928 06:45:38.467064 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:38.467071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:38.467104 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:45:38.467163 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:38.467166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:38.467218 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:38.467313 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:38.467640 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.468177 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.469243 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.469632 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.469751 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.469814 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:38.469991 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 06:45:38.470061 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:45:38.470376 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.466819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:43311 (local address 127.7.48.3:47849)
0928 06:45:38.467251 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:38.467263 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:38.467411 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:38.467691 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:38.467694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:38.467708 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:38.467817 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:38.467822 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.468539 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.468542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:38.469280 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:38.469288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.469948 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.469952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:38.470060 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.470077 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 06:45:38.470170 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 06:45:38.470237 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":314,"spinlock_wait_cycles":896}
I20260928 06:45:38.470371 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.466938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:47849 (local address 127.7.48.2:43311)
0928 06:45:38.467078 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:38.467082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:38.467367 (+ 285us) server_negotiation.cc:407] Connection header received
0928 06:45:38.467526 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:38.467529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:38.467584 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:38.467682 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:38.467955 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.468425 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.469403 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.469825 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.469945 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.470002 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:38.470185 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 06:45:38.470254 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 06:45:38.471012 14402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a75e4f057e44d8b20e510bcd141305" candidate_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99dc5bf901764934b1e36eeeb2d1b276" is_pre_election: true
I20260928 06:45:38.471215 14539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a75e4f057e44d8b20e510bcd141305" candidate_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b0b1b727b634600a687052b21a952b2" is_pre_election: true
I20260928 06:45:38.471372 14402 raft_consensus.cc:2454] T 45a75e4f057e44d8b20e510bcd141305 P 99dc5bf901764934b1e36eeeb2d1b276 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f84f119cbe8f49f5b7d2c0fd0009a3b6 in term 0.
I20260928 06:45:38.471470 14539 raft_consensus.cc:2454] T 45a75e4f057e44d8b20e510bcd141305 P 3b0b1b727b634600a687052b21a952b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f84f119cbe8f49f5b7d2c0fd0009a3b6 in term 0.
I20260928 06:45:38.471943 14610 leader_election.cc:302] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3b0b1b727b634600a687052b21a952b2, 99dc5bf901764934b1e36eeeb2d1b276, f84f119cbe8f49f5b7d2c0fd0009a3b6; no voters:
I20260928 06:45:38.472239 14762 raft_consensus.cc:2781] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:45:38.472316 14762 raft_consensus.cc:482] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:45:38.472345 14762 raft_consensus.cc:3037] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:38.473464 14762 raft_consensus.cc:504] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99dc5bf901764934b1e36eeeb2d1b276" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35937 } } peers { permanent_uuid: "3b0b1b727b634600a687052b21a952b2" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 43311 } } peers { permanent_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41911 } }
I20260928 06:45:38.473860 14762 leader_election.cc:288] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6 [CANDIDATE]: Term 1 election: Requested vote from peers 99dc5bf901764934b1e36eeeb2d1b276 (127.7.48.1:35937), 3b0b1b727b634600a687052b21a952b2 (127.7.48.2:43311)
I20260928 06:45:38.474210 14539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a75e4f057e44d8b20e510bcd141305" candidate_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b0b1b727b634600a687052b21a952b2"
I20260928 06:45:38.474210 14402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a75e4f057e44d8b20e510bcd141305" candidate_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99dc5bf901764934b1e36eeeb2d1b276"
I20260928 06:45:38.474337 14539 raft_consensus.cc:3037] T 45a75e4f057e44d8b20e510bcd141305 P 3b0b1b727b634600a687052b21a952b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:38.474337 14402 raft_consensus.cc:3037] T 45a75e4f057e44d8b20e510bcd141305 P 99dc5bf901764934b1e36eeeb2d1b276 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:38.475653 14402 raft_consensus.cc:2454] T 45a75e4f057e44d8b20e510bcd141305 P 99dc5bf901764934b1e36eeeb2d1b276 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f84f119cbe8f49f5b7d2c0fd0009a3b6 in term 1.
I20260928 06:45:38.475669 14539 raft_consensus.cc:2454] T 45a75e4f057e44d8b20e510bcd141305 P 3b0b1b727b634600a687052b21a952b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f84f119cbe8f49f5b7d2c0fd0009a3b6 in term 1.
I20260928 06:45:38.475970 14609 leader_election.cc:302] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b0b1b727b634600a687052b21a952b2, f84f119cbe8f49f5b7d2c0fd0009a3b6; no voters:
I20260928 06:45:38.476277 14762 raft_consensus.cc:2781] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:38.476521 14762 raft_consensus.cc:686] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6 [term 1 LEADER]: Becoming Leader. State: Replica: f84f119cbe8f49f5b7d2c0fd0009a3b6, State: Running, Role: LEADER
I20260928 06:45:38.476912 14762 consensus_queue.cc:237] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99dc5bf901764934b1e36eeeb2d1b276" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35937 } } peers { permanent_uuid: "3b0b1b727b634600a687052b21a952b2" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 43311 } } peers { permanent_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41911 } }
I20260928 06:45:38.480296 13477 catalog_manager.cc:5495] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f84f119cbe8f49f5b7d2c0fd0009a3b6 (127.7.48.3). New cstate: current_term: 1 leader_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99dc5bf901764934b1e36eeeb2d1b276" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b0b1b727b634600a687052b21a952b2" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41911 } health_report { overall_health: HEALTHY } } }
I20260928 06:45:38.504423 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.500791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.7.48.3:41911)
0928 06:45:38.500936 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:38.500939 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:38.500953 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:45:38.501078 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:38.501080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:38.501142 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:38.501230 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:45:38.501426 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.501905 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.502674 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.502899 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.502972 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.503276 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:45:38.503821 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:45:38.504060 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:38.504207 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 06:45:38.504269 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 06:45:38.510773 14539 raft_consensus.cc:1261] T 45a75e4f057e44d8b20e510bcd141305 P 3b0b1b727b634600a687052b21a952b2 [term 1 FOLLOWER]: Refusing update from remote peer f84f119cbe8f49f5b7d2c0fd0009a3b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:38.510805 14402 raft_consensus.cc:1261] T 45a75e4f057e44d8b20e510bcd141305 P 99dc5bf901764934b1e36eeeb2d1b276 [term 1 FOLLOWER]: Refusing update from remote peer f84f119cbe8f49f5b7d2c0fd0009a3b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:38.511523 14762 consensus_queue.cc:1035] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b0b1b727b634600a687052b21a952b2" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:38.511706 14764 consensus_queue.cc:1035] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99dc5bf901764934b1e36eeeb2d1b276" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:38.524590 14768 mvcc.cc:205] Tried to move back new op lower bound from 7334207236132995072 to 7334207236004069376. Current Snapshot: MvccSnapshot[applied={T|T < 7334207236132995072}]
I20260928 06:45:38.526082 14769 mvcc.cc:205] Tried to move back new op lower bound from 7334207236132995072 to 7334207236004069376. Current Snapshot: MvccSnapshot[applied={T|T < 7334207236132995072}]
I20260928 06:45:38.532655 13477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:38.532853 13477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:38.537642 13477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1137a312ab134aa4afef56800ac54199: Sending DeleteTablet for 3 replicas of tablet 45a75e4f057e44d8b20e510bcd141305
I20260928 06:45:38.538285 14382 tablet_service.cc:1514] Processing DeleteTablet for tablet 45a75e4f057e44d8b20e510bcd141305 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:38 UTC) from {username='oryx', principal='oryx/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:40342
I20260928 06:45:38.538388 14655 tablet_service.cc:1514] Processing DeleteTablet for tablet 45a75e4f057e44d8b20e510bcd141305 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:38 UTC) from {username='oryx', principal='oryx/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:43188
I20260928 06:45:38.538839 14777 tablet_replica.cc:330] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6: stopping tablet replica
I20260928 06:45:38.538843 14519 tablet_service.cc:1514] Processing DeleteTablet for tablet 45a75e4f057e44d8b20e510bcd141305 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:38 UTC) from {username='oryx', principal='oryx/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:36376
I20260928 06:45:38.539179 14777 raft_consensus.cc:2229] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:45:38.539372 14778 tablet_replica.cc:330] T 45a75e4f057e44d8b20e510bcd141305 P 3b0b1b727b634600a687052b21a952b2: stopping tablet replica
I20260928 06:45:38.539544 14777 raft_consensus.cc:2258] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:38.539937 14776 tablet_replica.cc:330] T 45a75e4f057e44d8b20e510bcd141305 P 99dc5bf901764934b1e36eeeb2d1b276: stopping tablet replica
I20260928 06:45:38.540218 14776 raft_consensus.cc:2229] T 45a75e4f057e44d8b20e510bcd141305 P 99dc5bf901764934b1e36eeeb2d1b276 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:38.540516 14776 raft_consensus.cc:2258] T 45a75e4f057e44d8b20e510bcd141305 P 99dc5bf901764934b1e36eeeb2d1b276 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:38.541411 14778 raft_consensus.cc:2229] T 45a75e4f057e44d8b20e510bcd141305 P 3b0b1b727b634600a687052b21a952b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:38.541755 14778 raft_consensus.cc:2258] T 45a75e4f057e44d8b20e510bcd141305 P 3b0b1b727b634600a687052b21a952b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:38.541937 14776 ts_tablet_manager.cc:1905] T 45a75e4f057e44d8b20e510bcd141305 P 99dc5bf901764934b1e36eeeb2d1b276: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:38.542325 14777 ts_tablet_manager.cc:1905] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:38.542372 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 13446
I20260928 06:45:38.543800 14778 ts_tablet_manager.cc:1905] T 45a75e4f057e44d8b20e510bcd141305 P 3b0b1b727b634600a687052b21a952b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:38.545373 14776 ts_tablet_manager.cc:1918] T 45a75e4f057e44d8b20e510bcd141305 P 99dc5bf901764934b1e36eeeb2d1b276: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:45:38.545475 14776 log.cc:1196] T 45a75e4f057e44d8b20e510bcd141305 P 99dc5bf901764934b1e36eeeb2d1b276: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-0/wal/wals/45a75e4f057e44d8b20e510bcd141305
I20260928 06:45:38.545789 14776 ts_tablet_manager.cc:1939] T 45a75e4f057e44d8b20e510bcd141305 P 99dc5bf901764934b1e36eeeb2d1b276: Deleting consensus metadata
I20260928 06:45:38.548704 14777 ts_tablet_manager.cc:1918] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:45:38.550850 14778 ts_tablet_manager.cc:1918] T 45a75e4f057e44d8b20e510bcd141305 P 3b0b1b727b634600a687052b21a952b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:45:38.551018 14777 log.cc:1196] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/45a75e4f057e44d8b20e510bcd141305
I20260928 06:45:38.551406 14777 ts_tablet_manager.cc:1939] T 45a75e4f057e44d8b20e510bcd141305 P f84f119cbe8f49f5b7d2c0fd0009a3b6: Deleting consensus metadata
W20260928 06:45:38.551614 14337 connection.cc:535] server connection from 127.0.0.1:40342 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:45:38.552814 14778 log.cc:1196] T 45a75e4f057e44d8b20e510bcd141305 P 3b0b1b727b634600a687052b21a952b2: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/45a75e4f057e44d8b20e510bcd141305
I20260928 06:45:38.553236 14778 ts_tablet_manager.cc:1939] T 45a75e4f057e44d8b20e510bcd141305 P 3b0b1b727b634600a687052b21a952b2: Deleting consensus metadata
W20260928 06:45:38.554406 14472 connection.cc:441] server connection from 127.0.0.1:36376 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36376 (request call id 1) could send its response
W20260928 06:45:38.554934 14608 connection.cc:441] server connection from 127.0.0.1:43188 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43188 (request call id 1) could send its response
I20260928 06:45:38.559063 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.541111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49422 (local address 127.7.48.61:33091)
0928 06:45:38.541326 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:38.541331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:38.549156 (+ 7825us) server_negotiation.cc:407] Connection header received
0928 06:45:38.549361 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:38.549364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:38.549425 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:38.549476 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:38.550484 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.551291 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.552941 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.553300 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.553370 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.555333 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:38.555337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:38.555340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:38.555367 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:38.557150 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.557664 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.557667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.557669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.557769 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.558101 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.558105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.558106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.558298 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:38.558393 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:38.558770 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 06:45:38.558911 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260928 06:45:38.565402 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.555585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49434 (local address 127.7.48.61:33091)
0928 06:45:38.555705 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:38.555709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:38.556570 (+ 861us) server_negotiation.cc:407] Connection header received
0928 06:45:38.556679 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:38.556682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:38.556721 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:38.556824 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:38.557546 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.558265 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.559596 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.559872 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.559968 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.561816 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:38.561819 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:38.561822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:38.561844 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:38.563450 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.563961 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.563967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.563971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.564056 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.564369 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.564374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.564377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.564605 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:38.564731 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:38.565117 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 06:45:38.565235 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:45:38.566131 14455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.556241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59574 (local address 127.7.48.60:40403)
0928 06:45:38.556376 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:38.556381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:38.557389 (+ 1008us) server_negotiation.cc:407] Connection header received
0928 06:45:38.557561 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:38.557564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:38.557610 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:38.557658 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:38.558449 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.559206 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.560681 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.561027 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.561102 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.562645 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:38.562649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:38.562651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:38.562676 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:38.564411 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.564883 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.564886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.564888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.564991 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.565271 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.565274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.565276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.565454 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:38.565550 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:38.565869 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 06:45:38.565987 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 06:45:38.568677 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:38.541216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59572 (local address 127.7.48.60:40403)
0928 06:45:38.541320 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:38.541326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:38.560436 (+ 19110us) server_negotiation.cc:407] Connection header received
0928 06:45:38.560602 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:38.560605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:38.560664 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:38.560747 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:38.561497 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.562279 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.563478 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:38.563855 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:38.564037 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:38.565592 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:38.565597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:38.565601 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:38.565629 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:38.567241 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.567598 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.567602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.567604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.567687 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:38.567945 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:38.567949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:38.567950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:38.568143 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:38.568261 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:38.568419 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 06:45:38.568531 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:45:39.525661 14786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:39.525208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.2:44401)
0928 06:45:39.525467 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:39.525558 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
W20260928 06:45:39.526067 14586 heartbeater.cc:644] Failed to heartbeat to 127.7.48.62:34479 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
I20260928 06:45:39.526769 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:39.526279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.3:35701)
0928 06:45:39.526591 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:39.526673 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":80,"threads_started":1}
W20260928 06:45:39.527066 14722 heartbeater.cc:644] Failed to heartbeat to 127.7.48.62:34479 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
I20260928 06:45:39.533702 14788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:39.533226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.1:34339)
0928 06:45:39.533489 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:39.533589 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
W20260928 06:45:39.533962 14449 heartbeater.cc:644] Failed to heartbeat to 127.7.48.62:34479 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
I20260928 06:45:40.037369 14789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1137a312ab134aa4afef56800ac54199)
I20260928 06:45:40.037621 14789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: false } }
I20260928 06:45:40.037843 14790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1137a312ab134aa4afef56800ac54199)
I20260928 06:45:40.038059 14790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: false } }
I20260928 06:45:40.038801 14789 leader_election.cc:288] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1137a312ab134aa4afef56800ac54199 (127.7.48.62:34479), 3c87976051a34263b3c105db63fb0fd0 (127.7.48.61:33091)
I20260928 06:45:40.038977 14790 leader_election.cc:288] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1137a312ab134aa4afef56800ac54199 (127.7.48.62:34479), 3ed6876b7d43478da762015583cd0eeb (127.7.48.60:40403)
I20260928 06:45:40.039492 14792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:40.038991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.0.0.1:42430)
0928 06:45:40.039293 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:40.039382 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":94,"threads_started":1}
W20260928 06:45:40.039758 14253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
I20260928 06:45:40.039803 14793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:40.039174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.0.0.1:42436)
0928 06:45:40.039634 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:40.039712 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":74,"threads_started":1}
W20260928 06:45:40.039999 14064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
I20260928 06:45:40.040781 14792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:40.040532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.0.0.1:42440)
0928 06:45:40.040669 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:40.040717 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 06:45:40.040887 14794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:40.040642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.0.0.1:42446)
0928 06:45:40.040769 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:40.040817 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260928 06:45:40.040942 14253 leader_election.cc:334] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1137a312ab134aa4afef56800ac54199 (127.7.48.62:34479): Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
W20260928 06:45:40.041066 14064 leader_election.cc:334] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1137a312ab134aa4afef56800ac54199 (127.7.48.62:34479): Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
I20260928 06:45:40.043537 14791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:40.039033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:33091 (local address 127.0.0.1:49436)
0928 06:45:40.039293 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:40.039310 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:40.039420 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:40.040222 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:40.040228 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:40.040255 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:40.040378 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:40.040387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:40.041491 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:40.041494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:40.042369 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:40.042377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:40.043081 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:40.043085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:40.043204 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:40.043230 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 06:45:40.043315 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 06:45:40.043404 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":4,"thread_start_us":63,"threads_started":1}
I20260928 06:45:40.043558 14795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:40.039446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49436 (local address 127.7.48.61:33091)
0928 06:45:40.039683 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:40.039689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:40.039707 (+ 18us) server_negotiation.cc:407] Connection header received
0928 06:45:40.039759 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:40.039762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:40.039817 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:40.040496 (+ 679us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:40.040610 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:40.041382 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:40.042486 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:40.042966 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:40.043049 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:40.043096 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:40.043351 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 06:45:40.043446 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":62,"threads_started":1}
I20260928 06:45:40.043846 14793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:40.039366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:40403 (local address 127.0.0.1:59576)
0928 06:45:40.039955 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:40.039971 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:40.040060 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:40.040666 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:40.040669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:40.040691 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:40.040784 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:40.040793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:40.041782 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:40.041784 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:40.042624 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:40.042632 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:40.043422 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:40.043425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:40.043506 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:40.043549 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 06:45:40.043630 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 06:45:40.043701 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":75,"threads_started":1}
I20260928 06:45:40.044073 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ed6876b7d43478da762015583cd0eeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3c87976051a34263b3c105db63fb0fd0" is_pre_election: true
I20260928 06:45:40.044306 14796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:40.039768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59576 (local address 127.7.48.60:40403)
0928 06:45:40.040403 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:40.040410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:40.040425 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:45:40.040472 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:40.040478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:40.040521 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:40.040898 (+ 377us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:40.041002 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:40.041677 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:40.042753 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:40.043318 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:40.043408 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:40.043469 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:40.043958 (+ 489us) server_negotiation.cc:299] Negotiation successful
0928 06:45:40.044159 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":577,"thread_start_us":57,"threads_started":1}
I20260928 06:45:40.044289 14090 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ed6876b7d43478da762015583cd0eeb in term 1.
I20260928 06:45:40.044726 14255 leader_election.cc:302] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c87976051a34263b3c105db63fb0fd0, 3ed6876b7d43478da762015583cd0eeb; no voters: 1137a312ab134aa4afef56800ac54199
I20260928 06:45:40.044759 14279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c87976051a34263b3c105db63fb0fd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3ed6876b7d43478da762015583cd0eeb" is_pre_election: true
I20260928 06:45:40.044937 14279 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c87976051a34263b3c105db63fb0fd0 in term 1.
I20260928 06:45:40.045045 14789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 06:45:40.045118 14789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1137a312ab134aa4afef56800ac54199)
I20260928 06:45:40.045176 14789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:45:40.045339 14066 leader_election.cc:302] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c87976051a34263b3c105db63fb0fd0, 3ed6876b7d43478da762015583cd0eeb; no voters: 1137a312ab134aa4afef56800ac54199
I20260928 06:45:40.045568 14790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 06:45:40.045640 14790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1137a312ab134aa4afef56800ac54199)
I20260928 06:45:40.045697 14790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:45:40.046355 14789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: false } }
I20260928 06:45:40.046720 14789 leader_election.cc:288] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [CANDIDATE]: Term 2 election: Requested vote from peers 1137a312ab134aa4afef56800ac54199 (127.7.48.62:34479), 3c87976051a34263b3c105db63fb0fd0 (127.7.48.61:33091)
I20260928 06:45:40.046717 14790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: false } }
I20260928 06:45:40.047106 14791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:40.046840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.0.0.1:42448)
0928 06:45:40.046995 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:40.047045 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 06:45:40.047049 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ed6876b7d43478da762015583cd0eeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3c87976051a34263b3c105db63fb0fd0"
I20260928 06:45:40.047137 14790 leader_election.cc:288] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [CANDIDATE]: Term 2 election: Requested vote from peers 1137a312ab134aa4afef56800ac54199 (127.7.48.62:34479), 3ed6876b7d43478da762015583cd0eeb (127.7.48.60:40403)
I20260928 06:45:40.047230 14090 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ed6876b7d43478da762015583cd0eeb in current term 2: Already voted for candidate 3c87976051a34263b3c105db63fb0fd0 in this term.
I20260928 06:45:40.047492 14793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:40.047226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.0.0.1:42456)
0928 06:45:40.047372 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:40.047423 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 06:45:40.047631 14279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c87976051a34263b3c105db63fb0fd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3ed6876b7d43478da762015583cd0eeb"
I20260928 06:45:40.047780 14279 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c87976051a34263b3c105db63fb0fd0 in current term 2: Already voted for candidate 3ed6876b7d43478da762015583cd0eeb in this term.
I20260928 06:45:40.047838 14791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:40.047641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.0.0.1:42470)
0928 06:45:40.047731 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:40.047783 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 06:45:40.047926 14253 leader_election.cc:334] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1137a312ab134aa4afef56800ac54199 (127.7.48.62:34479): Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
I20260928 06:45:40.047998 14253 leader_election.cc:302] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ed6876b7d43478da762015583cd0eeb; no voters: 1137a312ab134aa4afef56800ac54199, 3c87976051a34263b3c105db63fb0fd0
I20260928 06:45:40.048141 14789 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 06:45:40.048302 14793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:40.047977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.0.0.1:42474)
0928 06:45:40.048189 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:40.048242 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260928 06:45:40.048414 14064 leader_election.cc:334] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1137a312ab134aa4afef56800ac54199 (127.7.48.62:34479): Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
I20260928 06:45:40.048475 14064 leader_election.cc:302] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c87976051a34263b3c105db63fb0fd0; no voters: 1137a312ab134aa4afef56800ac54199, 3ed6876b7d43478da762015583cd0eeb
I20260928 06:45:40.048593 14790 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 06:45:40.527312 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:40.526782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.2:37951)
0928 06:45:40.527094 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:40.527160 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":121,"threads_started":1}
I20260928 06:45:40.528050 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:40.527628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.3:38775)
0928 06:45:40.527835 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:40.527957 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":70,"threads_started":1}
I20260928 06:45:40.534940 14801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:40.534448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.1:37941)
0928 06:45:40.534778 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:40.534843 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":75,"threads_started":1}
I20260928 06:45:41.528858 14802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:41.528331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.2:45083)
0928 06:45:41.528623 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:41.528687 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":80,"threads_started":1}
I20260928 06:45:41.529325 14803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:41.528954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.3:56511)
0928 06:45:41.529196 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:41.529248 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":86,"threads_started":1}
I20260928 06:45:41.535986 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:41.535548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.1:34527)
0928 06:45:41.535795 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:41.535848 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":85,"threads_started":1}
I20260928 06:45:41.903116 14805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:45:41.903262 14805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: false } }
I20260928 06:45:41.903780 14805 leader_election.cc:288] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1137a312ab134aa4afef56800ac54199 (127.7.48.62:34479), 3ed6876b7d43478da762015583cd0eeb (127.7.48.60:40403)
I20260928 06:45:41.904331 14279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c87976051a34263b3c105db63fb0fd0" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3ed6876b7d43478da762015583cd0eeb" is_pre_election: true
I20260928 06:45:41.904404 14806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:41.904032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.0.0.1:42704)
0928 06:45:41.904265 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:41.904320 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
I20260928 06:45:41.904469 14279 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c87976051a34263b3c105db63fb0fd0 in term 2.
I20260928 06:45:41.904816 14066 leader_election.cc:302] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c87976051a34263b3c105db63fb0fd0, 3ed6876b7d43478da762015583cd0eeb; no voters:
I20260928 06:45:41.905018 14805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260928 06:45:41.905094 14805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:45:41.905148 14805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 2 FOLLOWER]: Advancing to term 3
I20260928 06:45:41.905331 14806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:41.905061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.0.0.1:42716)
0928 06:45:41.905203 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:41.905270 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 06:45:41.905431 14064 leader_election.cc:334] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1137a312ab134aa4afef56800ac54199 (127.7.48.62:34479): Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
I20260928 06:45:41.906267 14805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: false } }
I20260928 06:45:41.906589 14805 leader_election.cc:288] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [CANDIDATE]: Term 3 election: Requested vote from peers 1137a312ab134aa4afef56800ac54199 (127.7.48.62:34479), 3ed6876b7d43478da762015583cd0eeb (127.7.48.60:40403)
I20260928 06:45:41.906956 14806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:41.906718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.0.0.1:42726)
0928 06:45:41.906828 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:41.906899 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 06:45:41.906973 14279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c87976051a34263b3c105db63fb0fd0" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3ed6876b7d43478da762015583cd0eeb"
I20260928 06:45:41.907100 14279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 2 FOLLOWER]: Advancing to term 3
I20260928 06:45:41.907675 14806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:41.907459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.0.0.1:42736)
0928 06:45:41.907576 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:41.907624 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 06:45:41.907825 14064 leader_election.cc:334] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1137a312ab134aa4afef56800ac54199 (127.7.48.62:34479): Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
I20260928 06:45:41.908344 14279 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c87976051a34263b3c105db63fb0fd0 in term 3.
I20260928 06:45:41.908674 14066 leader_election.cc:302] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c87976051a34263b3c105db63fb0fd0, 3ed6876b7d43478da762015583cd0eeb; no voters: 1137a312ab134aa4afef56800ac54199
I20260928 06:45:41.908840 14805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 06:45:41.909025 14805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [term 3 LEADER]: Becoming Leader. State: Replica: 3c87976051a34263b3c105db63fb0fd0, State: Running, Role: LEADER
I20260928 06:45:41.909307 14805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: false } }
I20260928 06:45:41.910802 14808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c87976051a34263b3c105db63fb0fd0. Latest consensus state: current_term: 3 leader_uuid: "3c87976051a34263b3c105db63fb0fd0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: false } } }
I20260928 06:45:41.910909 14808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:41.911242 14810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:45:41.913594 14810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:45:41.914038 14810 catalog_manager.cc:1259] Loaded cluster ID: 5736c5ca81164407b939a2e06709be0c
I20260928 06:45:41.914114 14810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:45:41.914665 14810 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:45:41.915050 14810 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0: Loaded TSK: 0
I20260928 06:45:41.915740 14810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:45:41.939613 14080 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:45:41.942826 14080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:45:41.946044 14806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:41.945732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.0.0.1:42758)
0928 06:45:41.945881 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:41.945957 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 06:45:41.946067 14279 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [term 3 FOLLOWER]: Refusing update from remote peer 3c87976051a34263b3c105db63fb0fd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260928 06:45:41.946558 14808 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 06:45:41.946841 14806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:41.946626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.0.0.1:42770)
0928 06:45:41.946735 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:41.946786 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 06:45:41.946959 14064 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 -> Peer 1137a312ab134aa4afef56800ac54199 (127.7.48.62:34479): Couldn't send request to peer 1137a312ab134aa4afef56800ac54199. Status: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 06:45:41.948115 14812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c87976051a34263b3c105db63fb0fd0. Latest consensus state: current_term: 3 leader_uuid: "3c87976051a34263b3c105db63fb0fd0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: false } } }
I20260928 06:45:41.948247 14812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:45:41.949285 14805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "3c87976051a34263b3c105db63fb0fd0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: false } } }
I20260928 06:45:41.949414 14805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0 [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:41.949697 14812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "3c87976051a34263b3c105db63fb0fd0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1137a312ab134aa4afef56800ac54199" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34479 } } peers { permanent_uuid: "3c87976051a34263b3c105db63fb0fd0" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "3ed6876b7d43478da762015583cd0eeb" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 40403 } attrs { promote: false } } }
I20260928 06:45:41.949831 14812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed6876b7d43478da762015583cd0eeb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:45:41.961151 14818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:41.956797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40344 (local address 127.7.48.1:35937)
0928 06:45:41.957071 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:41.957086 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:41.957101 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:45:41.957149 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:41.957152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:41.957190 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:41.957275 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:41.957557 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:41.958342 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:41.959926 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:41.960557 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:41.960669 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:41.960739 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:41.960931 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 06:45:41.961002 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":100,"threads_started":1}
I20260928 06:45:41.961127 14806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:41.956217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:35937 (local address 127.0.0.1:40344)
0928 06:45:41.956351 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:41.956367 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:41.956464 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:41.957292 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:41.957295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:41.957313 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:41.957430 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:41.957437 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:41.958459 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:41.958463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:41.959773 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:41.959781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:41.960688 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:41.960692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:41.960789 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:41.960817 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 06:45:41.960906 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:45:41.960988 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":46}
I20260928 06:45:41.961912 14382 tablet_service.cc:1467] Processing CreateTablet for tablet a31a528b44bd44c8abf8f2a7473d6f7f (DEFAULT_TABLE table=test-table [id=d2b04f9dc9214dc9b0500d43879a9230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:41.962252 14382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a31a528b44bd44c8abf8f2a7473d6f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:41.962246 14817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:41.957849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:43311 (local address 127.0.0.1:36390)
0928 06:45:41.958129 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:41.958144 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:41.958229 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:41.958760 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:41.958763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:41.958778 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:41.958864 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:41.958869 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:41.959735 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:41.959738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:41.960848 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:41.960859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:41.961826 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:41.961830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:41.961910 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:41.961948 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 06:45:41.962024 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:45:41.962096 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":84,"thread_start_us":95,"threads_started":1}
I20260928 06:45:41.962435 14820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:41.956431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:41911 (local address 127.0.0.1:43194)
0928 06:45:41.958082 (+ 1651us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:41.958098 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:41.958196 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:41.958559 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:41.958562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:41.958577 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:41.958717 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:41.958725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:41.959744 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:41.959748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:41.961096 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:41.961107 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:41.962016 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:41.962020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:41.962120 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:41.962141 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 06:45:41.962239 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 06:45:41.962308 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1584,"mutex_wait_us":76,"thread_start_us":99,"threads_started":1}
I20260928 06:45:41.962445 14819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:41.957077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43194 (local address 127.7.48.3:41911)
0928 06:45:41.957644 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:41.957648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:41.958265 (+ 617us) server_negotiation.cc:407] Connection header received
0928 06:45:41.958317 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:41.958319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:41.958358 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:41.958449 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:41.958852 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:41.959615 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:41.961241 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:41.961882 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:41.962005 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:41.962068 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:41.962255 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 06:45:41.962328 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":77,"threads_started":1}
I20260928 06:45:41.962636 14821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:41.957976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36390 (local address 127.7.48.2:43311)
0928 06:45:41.958549 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:41.958552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:41.958562 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:45:41.958606 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:41.958608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:41.958644 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:41.958753 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:41.958991 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:41.959613 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:41.960990 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:41.961569 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:41.961700 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:41.961766 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:41.962392 (+ 626us) server_negotiation.cc:299] Negotiation successful
0928 06:45:41.962478 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":101,"threads_started":1}
I20260928 06:45:41.962937 14655 tablet_service.cc:1467] Processing CreateTablet for tablet a31a528b44bd44c8abf8f2a7473d6f7f (DEFAULT_TABLE table=test-table [id=d2b04f9dc9214dc9b0500d43879a9230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:41.963223 14655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a31a528b44bd44c8abf8f2a7473d6f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:41.964609 14822 tablet_bootstrap.cc:492] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276: Bootstrap starting.
I20260928 06:45:41.965592 14822 tablet_bootstrap.cc:654] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:41.965642 14823 tablet_bootstrap.cc:492] T a31a528b44bd44c8abf8f2a7473d6f7f P f84f119cbe8f49f5b7d2c0fd0009a3b6: Bootstrap starting.
I20260928 06:45:41.965637 14519 tablet_service.cc:1467] Processing CreateTablet for tablet a31a528b44bd44c8abf8f2a7473d6f7f (DEFAULT_TABLE table=test-table [id=d2b04f9dc9214dc9b0500d43879a9230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:41.965920 14519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a31a528b44bd44c8abf8f2a7473d6f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:41.966571 14823 tablet_bootstrap.cc:654] T a31a528b44bd44c8abf8f2a7473d6f7f P f84f119cbe8f49f5b7d2c0fd0009a3b6: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:41.967077 14822 tablet_bootstrap.cc:492] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276: No bootstrap required, opened a new log
I20260928 06:45:41.967166 14822 ts_tablet_manager.cc:1397] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:45:41.967643 14822 raft_consensus.cc:348] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99dc5bf901764934b1e36eeeb2d1b276" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35937 } } peers { permanent_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41911 } } peers { permanent_uuid: "3b0b1b727b634600a687052b21a952b2" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 43311 } }
I20260928 06:45:41.967748 14822 raft_consensus.cc:374] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:41.967779 14822 raft_consensus.cc:729] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99dc5bf901764934b1e36eeeb2d1b276, State: Initialized, Role: FOLLOWER
I20260928 06:45:41.967955 14822 consensus_queue.cc:260] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99dc5bf901764934b1e36eeeb2d1b276" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35937 } } peers { permanent_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41911 } } peers { permanent_uuid: "3b0b1b727b634600a687052b21a952b2" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 43311 } }
I20260928 06:45:41.968362 14822 ts_tablet_manager.cc:1428] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 06:45:41.968528 14827 tablet_bootstrap.cc:492] T a31a528b44bd44c8abf8f2a7473d6f7f P 3b0b1b727b634600a687052b21a952b2: Bootstrap starting.
I20260928 06:45:41.968822 14823 tablet_bootstrap.cc:492] T a31a528b44bd44c8abf8f2a7473d6f7f P f84f119cbe8f49f5b7d2c0fd0009a3b6: No bootstrap required, opened a new log
I20260928 06:45:41.968910 14823 ts_tablet_manager.cc:1397] T a31a528b44bd44c8abf8f2a7473d6f7f P f84f119cbe8f49f5b7d2c0fd0009a3b6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:45:41.969373 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:41.969087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.1:40119)
0928 06:45:41.969242 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:41.969299 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 06:45:41.969326 14823 raft_consensus.cc:348] T a31a528b44bd44c8abf8f2a7473d6f7f P f84f119cbe8f49f5b7d2c0fd0009a3b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99dc5bf901764934b1e36eeeb2d1b276" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35937 } } peers { permanent_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41911 } } peers { permanent_uuid: "3b0b1b727b634600a687052b21a952b2" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 43311 } }
I20260928 06:45:41.969446 14823 raft_consensus.cc:374] T a31a528b44bd44c8abf8f2a7473d6f7f P f84f119cbe8f49f5b7d2c0fd0009a3b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:41.969488 14823 raft_consensus.cc:729] T a31a528b44bd44c8abf8f2a7473d6f7f P f84f119cbe8f49f5b7d2c0fd0009a3b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f84f119cbe8f49f5b7d2c0fd0009a3b6, State: Initialized, Role: FOLLOWER
I20260928 06:45:41.969532 14827 tablet_bootstrap.cc:654] T a31a528b44bd44c8abf8f2a7473d6f7f P 3b0b1b727b634600a687052b21a952b2: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:41.969623 14823 consensus_queue.cc:260] T a31a528b44bd44c8abf8f2a7473d6f7f P f84f119cbe8f49f5b7d2c0fd0009a3b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99dc5bf901764934b1e36eeeb2d1b276" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35937 } } peers { permanent_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41911 } } peers { permanent_uuid: "3b0b1b727b634600a687052b21a952b2" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 43311 } }
I20260928 06:45:41.969992 14823 ts_tablet_manager.cc:1428] T a31a528b44bd44c8abf8f2a7473d6f7f P f84f119cbe8f49f5b7d2c0fd0009a3b6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:45:41.970584 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:41.970344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.1:59727)
0928 06:45:41.970480 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:41.970527 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 06:45:41.971163 14803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:41.970876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.3:50557)
0928 06:45:41.971045 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:41.971095 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260928 06:45:41.972151 14450 heartbeater.cc:503] Master 127.7.48.61:33091 was elected leader, sending a full tablet report...
I20260928 06:45:41.972666 14723 heartbeater.cc:503] Master 127.7.48.61:33091 was elected leader, sending a full tablet report...
I20260928 06:45:41.973155 14803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:41.972876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.3:47053)
0928 06:45:41.973016 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:41.973084 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 06:45:41.974148 14827 tablet_bootstrap.cc:492] T a31a528b44bd44c8abf8f2a7473d6f7f P 3b0b1b727b634600a687052b21a952b2: No bootstrap required, opened a new log
I20260928 06:45:41.974259 14827 ts_tablet_manager.cc:1397] T a31a528b44bd44c8abf8f2a7473d6f7f P 3b0b1b727b634600a687052b21a952b2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 06:45:41.974763 14827 raft_consensus.cc:348] T a31a528b44bd44c8abf8f2a7473d6f7f P 3b0b1b727b634600a687052b21a952b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99dc5bf901764934b1e36eeeb2d1b276" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35937 } } peers { permanent_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41911 } } peers { permanent_uuid: "3b0b1b727b634600a687052b21a952b2" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 43311 } }
I20260928 06:45:41.974860 14827 raft_consensus.cc:374] T a31a528b44bd44c8abf8f2a7473d6f7f P 3b0b1b727b634600a687052b21a952b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:41.974882 14827 raft_consensus.cc:729] T a31a528b44bd44c8abf8f2a7473d6f7f P 3b0b1b727b634600a687052b21a952b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b0b1b727b634600a687052b21a952b2, State: Initialized, Role: FOLLOWER
I20260928 06:45:41.975037 14827 consensus_queue.cc:260] T a31a528b44bd44c8abf8f2a7473d6f7f P 3b0b1b727b634600a687052b21a952b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99dc5bf901764934b1e36eeeb2d1b276" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35937 } } peers { permanent_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41911 } } peers { permanent_uuid: "3b0b1b727b634600a687052b21a952b2" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 43311 } }
I20260928 06:45:41.975417 14827 ts_tablet_manager.cc:1428] T a31a528b44bd44c8abf8f2a7473d6f7f P 3b0b1b727b634600a687052b21a952b2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 06:45:41.976059 14802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:41.975773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.2:58157)
0928 06:45:41.975914 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:41.975988 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 06:45:41.976797 14587 heartbeater.cc:503] Master 127.7.48.61:33091 was elected leader, sending a full tablet report...
I20260928 06:45:41.977065 14802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:41.976820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.2:54115)
0928 06:45:41.976948 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:41.977003 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 06:45:42.104120 14724 tablet.cc:2369] T a31a528b44bd44c8abf8f2a7473d6f7f P f84f119cbe8f49f5b7d2c0fd0009a3b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:45:42.157584 14588 tablet.cc:2369] T a31a528b44bd44c8abf8f2a7473d6f7f P 3b0b1b727b634600a687052b21a952b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:45:42.200564 14826 raft_consensus.cc:482] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:45:42.200763 14826 raft_consensus.cc:504] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99dc5bf901764934b1e36eeeb2d1b276" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35937 } } peers { permanent_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41911 } } peers { permanent_uuid: "3b0b1b727b634600a687052b21a952b2" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 43311 } }
I20260928 06:45:42.202008 14826 leader_election.cc:288] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f84f119cbe8f49f5b7d2c0fd0009a3b6 (127.7.48.3:41911), 3b0b1b727b634600a687052b21a952b2 (127.7.48.2:43311)
I20260928 06:45:42.205830 14819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:42.202439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:44521 (local address 127.7.48.3:41911)
0928 06:45:42.202650 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:42.202656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:42.202681 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:45:42.202737 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:42.202739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:42.202783 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:42.202884 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:42.203177 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:42.203684 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:42.204801 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:42.205202 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:42.205327 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:42.205402 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:42.205632 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 06:45:42.205699 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260928 06:45:42.205850 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:42.202216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:41911 (local address 127.7.48.1:44521)
0928 06:45:42.202352 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:42.202371 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:42.202497 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:42.202899 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:42.202902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:42.202921 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:42.203056 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:42.203062 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:42.203804 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:42.203807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:42.204667 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:42.204677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:42.205356 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:42.205360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:42.205481 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:42.205500 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 06:45:42.205611 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 06:45:42.205686 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 06:45:42.205901 14821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:42.202465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:54511 (local address 127.7.48.2:43311)
0928 06:45:42.202607 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:42.202612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:42.202827 (+ 215us) server_negotiation.cc:407] Connection header received
0928 06:45:42.202983 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:42.202991 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:42.203033 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:42.203122 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:42.203382 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:42.203865 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:42.204918 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:42.205320 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:42.205442 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:42.205503 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:42.205698 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 06:45:42.205772 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:45:42.206008 14831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:42.202339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:43311 (local address 127.7.48.1:54511)
0928 06:45:42.202732 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:42.202743 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:42.202861 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:42.203137 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:42.203140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:42.203156 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:42.203270 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:42.203275 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:42.204035 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:42.204038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:42.204786 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:42.204795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:42.205448 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:42.205452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:42.205571 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:42.205589 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 06:45:42.205683 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 06:45:42.205753 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":145,"threads_started":1}
I20260928 06:45:42.206485 14539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a31a528b44bd44c8abf8f2a7473d6f7f" candidate_uuid: "99dc5bf901764934b1e36eeeb2d1b276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b0b1b727b634600a687052b21a952b2" is_pre_election: true
I20260928 06:45:42.206517 14675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a31a528b44bd44c8abf8f2a7473d6f7f" candidate_uuid: "99dc5bf901764934b1e36eeeb2d1b276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" is_pre_election: true
I20260928 06:45:42.206653 14539 raft_consensus.cc:2454] T a31a528b44bd44c8abf8f2a7473d6f7f P 3b0b1b727b634600a687052b21a952b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99dc5bf901764934b1e36eeeb2d1b276 in term 0.
I20260928 06:45:42.206737 14675 raft_consensus.cc:2454] T a31a528b44bd44c8abf8f2a7473d6f7f P f84f119cbe8f49f5b7d2c0fd0009a3b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99dc5bf901764934b1e36eeeb2d1b276 in term 0.
I20260928 06:45:42.207073 14336 leader_election.cc:302] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b0b1b727b634600a687052b21a952b2, 99dc5bf901764934b1e36eeeb2d1b276; no voters:
I20260928 06:45:42.207399 14826 raft_consensus.cc:2781] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:45:42.207468 14826 raft_consensus.cc:482] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:45:42.207508 14826 raft_consensus.cc:3037] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:42.208563 14826 raft_consensus.cc:504] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99dc5bf901764934b1e36eeeb2d1b276" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35937 } } peers { permanent_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41911 } } peers { permanent_uuid: "3b0b1b727b634600a687052b21a952b2" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 43311 } }
I20260928 06:45:42.208925 14826 leader_election.cc:288] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276 [CANDIDATE]: Term 1 election: Requested vote from peers f84f119cbe8f49f5b7d2c0fd0009a3b6 (127.7.48.3:41911), 3b0b1b727b634600a687052b21a952b2 (127.7.48.2:43311)
I20260928 06:45:42.209235 14675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a31a528b44bd44c8abf8f2a7473d6f7f" candidate_uuid: "99dc5bf901764934b1e36eeeb2d1b276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6"
I20260928 06:45:42.209292 14539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a31a528b44bd44c8abf8f2a7473d6f7f" candidate_uuid: "99dc5bf901764934b1e36eeeb2d1b276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b0b1b727b634600a687052b21a952b2"
I20260928 06:45:42.209384 14675 raft_consensus.cc:3037] T a31a528b44bd44c8abf8f2a7473d6f7f P f84f119cbe8f49f5b7d2c0fd0009a3b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:42.209478 14539 raft_consensus.cc:3037] T a31a528b44bd44c8abf8f2a7473d6f7f P 3b0b1b727b634600a687052b21a952b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:42.210386 14539 raft_consensus.cc:2454] T a31a528b44bd44c8abf8f2a7473d6f7f P 3b0b1b727b634600a687052b21a952b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99dc5bf901764934b1e36eeeb2d1b276 in term 1.
I20260928 06:45:42.210402 14675 raft_consensus.cc:2454] T a31a528b44bd44c8abf8f2a7473d6f7f P f84f119cbe8f49f5b7d2c0fd0009a3b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99dc5bf901764934b1e36eeeb2d1b276 in term 1.
I20260928 06:45:42.210680 14336 leader_election.cc:302] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b0b1b727b634600a687052b21a952b2, 99dc5bf901764934b1e36eeeb2d1b276; no voters:
I20260928 06:45:42.210947 14826 raft_consensus.cc:2781] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:42.211141 14826 raft_consensus.cc:686] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276 [term 1 LEADER]: Becoming Leader. State: Replica: 99dc5bf901764934b1e36eeeb2d1b276, State: Running, Role: LEADER
I20260928 06:45:42.211413 14826 consensus_queue.cc:237] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99dc5bf901764934b1e36eeeb2d1b276" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35937 } } peers { permanent_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41911 } } peers { permanent_uuid: "3b0b1b727b634600a687052b21a952b2" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 43311 } }
I20260928 06:45:42.211858 14831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:42.211634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.1:46237)
0928 06:45:42.211735 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:42.211791 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 06:45:42.214051 14079 catalog_manager.cc:5495] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99dc5bf901764934b1e36eeeb2d1b276 (127.7.48.1). New cstate: current_term: 1 leader_uuid: "99dc5bf901764934b1e36eeeb2d1b276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99dc5bf901764934b1e36eeeb2d1b276" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b0b1b727b634600a687052b21a952b2" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 43311 } health_report { overall_health: UNKNOWN } } }
I20260928 06:45:42.254653 14818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:42.251206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40354 (local address 127.7.48.1:35937)
0928 06:45:42.251341 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:42.251345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:42.251357 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:45:42.251436 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:42.251438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:42.251494 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:42.251585 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:45:42.251822 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:42.252364 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:42.253239 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:42.253477 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:42.253564 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:42.253810 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:45:42.254320 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:45:42.254416 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:42.254478 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 06:45:42.254528 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 06:45:42.257931 14539 raft_consensus.cc:1261] T a31a528b44bd44c8abf8f2a7473d6f7f P 3b0b1b727b634600a687052b21a952b2 [term 1 FOLLOWER]: Refusing update from remote peer 99dc5bf901764934b1e36eeeb2d1b276: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:42.258119 14675 raft_consensus.cc:1261] T a31a528b44bd44c8abf8f2a7473d6f7f P f84f119cbe8f49f5b7d2c0fd0009a3b6 [term 1 FOLLOWER]: Refusing update from remote peer 99dc5bf901764934b1e36eeeb2d1b276: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:42.258504 14826 consensus_queue.cc:1035] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b0b1b727b634600a687052b21a952b2" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:42.258719 14832 consensus_queue.cc:1035] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f84f119cbe8f49f5b7d2c0fd0009a3b6" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:42.260499 14802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:42.260206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.2:39737)
0928 06:45:42.260364 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:42.260422 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 06:45:42.261914 14803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:42.261629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.3:53745)
0928 06:45:42.261750 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:42.261837 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 06:45:42.265815 14831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:42.265549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.1:59327)
0928 06:45:42.265676 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:42.265738 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 06:45:42.266364 14802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:42.266035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.2:52985)
0928 06:45:42.266231 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:42.266285 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260928 06:45:42.266804 14838 mvcc.cc:205] Tried to move back new op lower bound from 7334207251482849280 to 7334207251301081088. Current Snapshot: MvccSnapshot[applied={T|T < 7334207251482849280 or (T in {7334207251482849280})}]
I20260928 06:45:42.267729 14803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:42.266826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34479 (local address 127.7.48.3:37199)
0928 06:45:42.267599 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:42.267650 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":699}
I20260928 06:45:42.274139 14080 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:42.274322 14080 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:42.279479 14080 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c87976051a34263b3c105db63fb0fd0: Sending DeleteTablet for 3 replicas of tablet a31a528b44bd44c8abf8f2a7473d6f7f
I20260928 06:45:42.280455 14382 tablet_service.cc:1514] Processing DeleteTablet for tablet a31a528b44bd44c8abf8f2a7473d6f7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:42 UTC) from {username='oryx', principal='oryx/127.7.48.61@KRBTEST.COM'} at 127.0.0.1:40344
I20260928 06:45:42.280479 14655 tablet_service.cc:1514] Processing DeleteTablet for tablet a31a528b44bd44c8abf8f2a7473d6f7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:42 UTC) from {username='oryx', principal='oryx/127.7.48.61@KRBTEST.COM'} at 127.0.0.1:43194
I20260928 06:45:42.280864 14846 tablet_replica.cc:330] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276: stopping tablet replica
I20260928 06:45:42.280987 14846 raft_consensus.cc:2229] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:45:42.281277 14519 tablet_service.cc:1514] Processing DeleteTablet for tablet a31a528b44bd44c8abf8f2a7473d6f7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:42 UTC) from {username='oryx', principal='oryx/127.7.48.61@KRBTEST.COM'} at 127.0.0.1:36390
I20260928 06:45:42.281325 14846 raft_consensus.cc:2258] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:42.281629 14847 tablet_replica.cc:330] T a31a528b44bd44c8abf8f2a7473d6f7f P f84f119cbe8f49f5b7d2c0fd0009a3b6: stopping tablet replica
I20260928 06:45:42.281808 14847 raft_consensus.cc:2229] T a31a528b44bd44c8abf8f2a7473d6f7f P f84f119cbe8f49f5b7d2c0fd0009a3b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:42.282034 14848 tablet_replica.cc:330] T a31a528b44bd44c8abf8f2a7473d6f7f P 3b0b1b727b634600a687052b21a952b2: stopping tablet replica
I20260928 06:45:42.282195 14848 raft_consensus.cc:2229] T a31a528b44bd44c8abf8f2a7473d6f7f P 3b0b1b727b634600a687052b21a952b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:42.282197 14846 ts_tablet_manager.cc:1905] T a31a528b44bd44c8abf8f2a7473d6f7f P 99dc5bf901764934b1e36eeeb2d1b276: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:42.282356 14848 raft_consensus.cc:2258] T a31a528b44bd44c8abf8f2a7473d6f7f P 3b0b1b727b634600a687052b21a952b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:42.282945 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 14224
I20260928 06:45:42.284691 14847 raft_consensus.cc:2258] T a31a528b44bd44c8abf8f2a7473d6f7f P f84f119cbe8f49f5b7d2c0fd0009a3b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:42.285398 14847 ts_tablet_manager.cc:1905] T a31a528b44bd44c8abf8f2a7473d6f7f P f84f119cbe8f49f5b7d2c0fd0009a3b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:42.285552 14848 ts_tablet_manager.cc:1905] T a31a528b44bd44c8abf8f2a7473d6f7f P 3b0b1b727b634600a687052b21a952b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:42.288092 14847 ts_tablet_manager.cc:1918] T a31a528b44bd44c8abf8f2a7473d6f7f P f84f119cbe8f49f5b7d2c0fd0009a3b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:45:42.288190 14847 log.cc:1196] T a31a528b44bd44c8abf8f2a7473d6f7f P f84f119cbe8f49f5b7d2c0fd0009a3b6: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/a31a528b44bd44c8abf8f2a7473d6f7f
I20260928 06:45:42.288514 14847 ts_tablet_manager.cc:1939] T a31a528b44bd44c8abf8f2a7473d6f7f P f84f119cbe8f49f5b7d2c0fd0009a3b6: Deleting consensus metadata
I20260928 06:45:42.288911 14848 ts_tablet_manager.cc:1918] T a31a528b44bd44c8abf8f2a7473d6f7f P 3b0b1b727b634600a687052b21a952b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:45:42.288993 14848 log.cc:1196] T a31a528b44bd44c8abf8f2a7473d6f7f P 3b0b1b727b634600a687052b21a952b2: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/a31a528b44bd44c8abf8f2a7473d6f7f
I20260928 06:45:42.289304 14848 ts_tablet_manager.cc:1939] T a31a528b44bd44c8abf8f2a7473d6f7f P 3b0b1b727b634600a687052b21a952b2: Deleting consensus metadata
I20260928 06:45:42.289273 14066 catalog_manager.cc:4841] TS f84f119cbe8f49f5b7d2c0fd0009a3b6 (127.7.48.3:41911): tablet a31a528b44bd44c8abf8f2a7473d6f7f (table test-table [id=d2b04f9dc9214dc9b0500d43879a9230]) successfully deleted
I20260928 06:45:42.290058 14065 catalog_manager.cc:4841] TS 3b0b1b727b634600a687052b21a952b2 (127.7.48.2:43311): tablet a31a528b44bd44c8abf8f2a7473d6f7f (table test-table [id=d2b04f9dc9214dc9b0500d43879a9230]) successfully deleted
I20260928 06:45:42.291039 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 14456
W20260928 06:45:42.291051 14820 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:42.290756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:35937 (local address 127.0.0.1:40356)
0928 06:45:42.290884 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:42.290940 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.1:35937: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 06:45:42.291285 14066 catalog_manager.cc:4584] TS 99dc5bf901764934b1e36eeeb2d1b276 (127.7.48.1:35937): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a31a528b44bd44c8abf8f2a7473d6f7f: Network error: Client connection negotiation failed: client connection to 127.7.48.1:35937: connect: Connection reset by peer (error 104)
I20260928 06:45:42.297366 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 14592
I20260928 06:45:42.304718 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 13942
I20260928 06:45:42.310664 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 14045
2026-09-28T06:45:42Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7297 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:45:42 dist-test-slave-twl6 krb5kdc[14851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:45:42 dist-test-slave-twl6 krb5kdc[14851](info): set up 2 sockets
Sep 28 06:45:42 dist-test-slave-twl6 krb5kdc[14851](info): commencing operation
krb5kdc: starting...
W20260928 06:45:43.345191 7360 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:45:43 dist-test-slave-twl6 krb5kdc[14851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:45:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:45:43Z Disabled control of system clock
WARNING: no policy specified for oryx/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.62.keytab.
Entry for principal oryx/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.62.keytab.
I20260928 06:45:43.473057 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:46109
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.62.keytab
--principal=oryx/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:46109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:43.587406 14864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:43.587693 14864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:43.587775 14864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:43.591816 14864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:45:43.591905 14864 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 06:45:43.591924 14864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:43.591985 14864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:43.592020 14864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:45:43.592043 14864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:45:43.597600 14864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:46109
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.62.keytab
--principal=oryx/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:46109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:43.598821 14864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:43.600028 14864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:43.608572 14870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:43.608588 14872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:43.608587 14869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:43.608788 14864 server_base.cc:1034] running on GCE node
I20260928 06:45:43.609475 14864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:43.610616 14864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:43.611814 14864 hybrid_clock.cc:648] HybridClock initialized: now 1790577943611788 us; error 43 us; skew 500 ppm
Sep 28 06:45:43 dist-test-slave-twl6 krb5kdc[14851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577943, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:43.614799 14864 init.cc:377] Logged in from keytab as oryx/127.7.48.62@KRBTEST.COM (short username oryx)
I20260928 06:45:43.615715 14864 webserver.cc:460] Webserver started at http://127.7.48.62:33399/ using document root <none> and password file <none>
I20260928 06:45:43.616348 14864 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:43.616415 14864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:43.616597 14864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:43.618695 14864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "34b336e939ad476ba98f1ae1063ffdcc"
format_stamp: "Formatted at 2026-09-28 06:45:43 on dist-test-slave-twl6"
server_key: "68cbca7ed77ae796d215b6c24735bf54"
server_key_iv: "23ab3f8a76cc3ed933d785d56ab15d31"
server_key_version: "encryptionkey@0"
I20260928 06:45:43.619300 14864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "34b336e939ad476ba98f1ae1063ffdcc"
format_stamp: "Formatted at 2026-09-28 06:45:43 on dist-test-slave-twl6"
server_key: "68cbca7ed77ae796d215b6c24735bf54"
server_key_iv: "23ab3f8a76cc3ed933d785d56ab15d31"
server_key_version: "encryptionkey@0"
I20260928 06:45:43.623281 14864 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 06:45:43.625885 14879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:43.627239 14864 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:43.627413 14864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "34b336e939ad476ba98f1ae1063ffdcc"
format_stamp: "Formatted at 2026-09-28 06:45:43 on dist-test-slave-twl6"
server_key: "68cbca7ed77ae796d215b6c24735bf54"
server_key_iv: "23ab3f8a76cc3ed933d785d56ab15d31"
server_key_version: "encryptionkey@0"
I20260928 06:45:43.627543 14864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:43.647636 14864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:43.651566 14864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:43.651737 14864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:43.659917 14864 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:46109
I20260928 06:45:43.659947 14941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:46109 every 8 connection(s)
I20260928 06:45:43.660902 14864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:45:43.663759 14942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:43.669411 14942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc: Bootstrap starting.
I20260928 06:45:43.670094 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 14864
I20260928 06:45:43.670205 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:45:43.670523 7360 external_mini_cluster.cc:1428] Setting key 42e1e054fd50cdbcf83f9ce86d1f957e
I20260928 06:45:43.672379 14942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:43.673133 14942 log.cc:824] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:43.675117 14942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc: No bootstrap required, opened a new log
I20260928 06:45:43.678591 14942 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b336e939ad476ba98f1ae1063ffdcc" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 46109 } }
I20260928 06:45:43.678862 14942 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:43.678943 14942 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34b336e939ad476ba98f1ae1063ffdcc, State: Initialized, Role: FOLLOWER
Sep 28 06:45:43 dist-test-slave-twl6 krb5kdc[14851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM
I20260928 06:45:43.679484 14942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b336e939ad476ba98f1ae1063ffdcc" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 46109 } }
I20260928 06:45:43.679677 14942 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:45:43.679744 14942 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:45:43.679839 14942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:43.680789 14942 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b336e939ad476ba98f1ae1063ffdcc" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 46109 } }
I20260928 06:45:43.681142 14942 leader_election.cc:302] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34b336e939ad476ba98f1ae1063ffdcc; no voters:
I20260928 06:45:43.681536 14942 leader_election.cc:288] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:45:43.681636 14947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:43.681933 14947 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc [term 1 LEADER]: Becoming Leader. State: Replica: 34b336e939ad476ba98f1ae1063ffdcc, State: Running, Role: LEADER
I20260928 06:45:43.682313 14947 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b336e939ad476ba98f1ae1063ffdcc" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 46109 } }
I20260928 06:45:43.682986 14942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:45:43.683633 14947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34b336e939ad476ba98f1ae1063ffdcc. Latest consensus state: current_term: 1 leader_uuid: "34b336e939ad476ba98f1ae1063ffdcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b336e939ad476ba98f1ae1063ffdcc" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 46109 } } }
I20260928 06:45:43.683842 14947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:43.684180 14948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34b336e939ad476ba98f1ae1063ffdcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b336e939ad476ba98f1ae1063ffdcc" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 46109 } } }
I20260928 06:45:43.684340 14948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:43.684433 14956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:45:43.686601 14945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:43.671811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56474 (local address 127.7.48.62:46109)
0928 06:45:43.672315 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:43.672330 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:43.672366 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:45:43.673078 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:43.673096 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:43.673535 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:43.673884 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:43.674733 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:43.675781 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:43.676667 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:43.677147 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:43.677328 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:43.680097 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:43.680102 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:43.680116 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:43.680144 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:43.682346 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:43.682891 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:43.682897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:43.682904 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:43.683067 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:43.683434 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:43.683437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:43.683439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:43.684088 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:43.684964 (+ 876us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:43.685161 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 06:45:43.685578 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":160,"threads_started":1}
I20260928 06:45:43.687922 14956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:45:43.695731 14956 catalog_manager.cc:1347] Generated new cluster ID: f2efcbfd57e24a3e93437c74652ceb55
I20260928 06:45:43.695816 14956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:45:43.706552 14956 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:45:43.707645 14956 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:45:43.712195 14956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc: Generated new TSK 0
I20260928 06:45:43.713035 14956 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.7.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.1.keytab.
Entry for principal oryx/127.7.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.1.keytab.
I20260928 06:45:43.748960 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:46109
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.1.keytab
--principal=oryx/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:43.877483 14968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:43.877760 14968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:43.877807 14968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:43.881814 14968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:43.881878 14968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:43.881918 14968 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:45:43.881953 14968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:45:43.887737 14968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.1.keytab
--principal=oryx/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.48.62:46109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:43.888868 14968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:43.889871 14968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:45:43.899756 14968 server_base.cc:1034] running on GCE node
W20260928 06:45:43.899780 14973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:43.899709 14974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:43.899709 14976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:43.900400 14968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:43.901057 14968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:43.902251 14968 hybrid_clock.cc:648] HybridClock initialized: now 1790577943902237 us; error 28 us; skew 500 ppm
Sep 28 06:45:43 dist-test-slave-twl6 krb5kdc[14851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577943, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:43.905108 14968 init.cc:377] Logged in from keytab as oryx/127.7.48.1@KRBTEST.COM (short username oryx)
I20260928 06:45:43.906041 14968 webserver.cc:460] Webserver started at http://127.7.48.1:37221/ using document root <none> and password file <none>
I20260928 06:45:43.906643 14968 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:43.906737 14968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:43.906991 14968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:43.909072 14968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "69f4f371eac34d12bca22a7bda991178"
format_stamp: "Formatted at 2026-09-28 06:45:43 on dist-test-slave-twl6"
server_key: "06a92c04885a96d3b0b0585b8a3e277b"
server_key_iv: "c2aa3696f6b1da162f390955b56df35a"
server_key_version: "encryptionkey@0"
I20260928 06:45:43.909643 14968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "69f4f371eac34d12bca22a7bda991178"
format_stamp: "Formatted at 2026-09-28 06:45:43 on dist-test-slave-twl6"
server_key: "06a92c04885a96d3b0b0585b8a3e277b"
server_key_iv: "c2aa3696f6b1da162f390955b56df35a"
server_key_version: "encryptionkey@0"
I20260928 06:45:43.913908 14968 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:45:43.916563 14983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:43.917761 14968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:45:43.917900 14968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "69f4f371eac34d12bca22a7bda991178"
format_stamp: "Formatted at 2026-09-28 06:45:43 on dist-test-slave-twl6"
server_key: "06a92c04885a96d3b0b0585b8a3e277b"
server_key_iv: "c2aa3696f6b1da162f390955b56df35a"
server_key_version: "encryptionkey@0"
I20260928 06:45:43.918017 14968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:43.944790 14968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:43.948580 14968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:43.948812 14968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:43.950380 14968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:43.950462 14968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:43.950536 14968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:43.950582 14968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:45:43 dist-test-slave-twl6 krb5kdc[14851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577943, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM
I20260928 06:45:43.962245 14968 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:36379
I20260928 06:45:43.962327 15098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:36379 every 8 connection(s)
I20260928 06:45:43.963296 14968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:45:43.965569 14945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:43.951511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:47007 (local address 127.7.48.62:46109)
0928 06:45:43.951673 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:43.951676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:43.952808 (+ 1132us) server_negotiation.cc:407] Connection header received
0928 06:45:43.954076 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:43.954079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:43.954141 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:43.954243 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:43.956113 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:43.957031 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:43.958563 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:43.958823 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:43.958945 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:43.961715 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:43.961719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:43.961722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:43.961752 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:43.963451 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:43.964093 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:43.964097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:43.964100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:43.964206 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:43.964567 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:43.964569 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:43.964570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:43.964759 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:43.964837 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:43.965313 (+ 476us) server_negotiation.cc:299] Negotiation successful
0928 06:45:43.965435 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 06:45:43.966419 14992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:43.952112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:46109 (local address 127.7.48.1:47007)
0928 06:45:43.952642 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:43.952687 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:43.953814 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:43.954486 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:43.954496 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:43.955031 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:43.955838 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:43.955864 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:43.957170 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:43.957175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:43.958425 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:43.958435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:43.958992 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:43.958996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:43.959203 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:43.959907 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:43.959929 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:43.961517 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:43.963546 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:43.963551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:43.963565 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:43.963955 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:43.964307 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:43.964310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:43.964311 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:43.964454 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:43.964850 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:43.964854 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:43.965083 (+ 229us) client_negotiation.cc:769] Sending connection context
0928 06:45:43.965236 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 06:45:43.965494 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":151,"threads_started":1}
I20260928 06:45:43.966470 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 14968
I20260928 06:45:43.966594 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
I20260928 06:45:43.966905 7360 external_mini_cluster.cc:1428] Setting key 2c83062ea270bcf99a9a7271a0140d51
I20260928 06:45:43.970144 15099 heartbeater.cc:347] Connected to a master server at 127.7.48.62:46109
I20260928 06:45:43.970304 15099 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:43.970999 15099 heartbeater.cc:511] Master 127.7.48.62:46109 requested a full tablet report, sending...
I20260928 06:45:43.972594 14896 ts_manager.cc:194] Registered new tserver with Master: 69f4f371eac34d12bca22a7bda991178 (127.7.48.1:36379)
I20260928 06:45:43.974304 14896 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.48.1@KRBTEST.COM'} at 127.7.48.1:47007
I20260928 06:45:43.978016 14945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:43.970782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:55845 (local address 127.7.48.62:46109)
0928 06:45:43.970922 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:43.970930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:43.970941 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:45:43.970971 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:43.970973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:43.971006 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:43.971098 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:43.971602 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:43.972254 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:43.973203 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:43.973437 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:43.973555 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:43.974777 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:43.974780 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:43.974783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:43.974809 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:43.976382 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:43.976821 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:43.976825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:43.976827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:43.976912 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:43.977206 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:43.977210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:43.977212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:43.977386 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:43.977500 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:43.977755 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 06:45:43.977866 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 06:45:43.978030 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:43.970321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:46109 (local address 127.7.48.1:55845)
0928 06:45:43.970625 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:43.970642 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:43.970735 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:43.971159 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:43.971163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:43.971368 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:43.971478 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:43.971484 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:43.972388 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:43.972391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:43.973069 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:43.973081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:43.973560 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:43.973563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:43.973656 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:43.974227 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:43.974239 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:43.974640 (+ 401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:43.976505 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:43.976509 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:43.976512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:43.976716 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:43.977015 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:43.977019 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:43.977021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:43.977110 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:43.977491 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:43.977494 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:43.977590 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 06:45:43.977701 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 06:45:43.977842 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for oryx/127.7.48.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.48.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.2.keytab.
Entry for principal oryx/127.7.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.2.keytab.
I20260928 06:45:43.999389 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:46109
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.2.keytab
--principal=oryx/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:44.117662 15108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:44.117949 15108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:44.117995 15108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:44.122352 15108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:44.122426 15108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:44.122468 15108 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:45:44.122502 15108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:45:44.128001 15108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.2.keytab
--principal=oryx/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.48.62:46109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:44.129057 15108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:44.130072 15108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:44.139927 15114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:44.139997 15108 server_base.cc:1034] running on GCE node
W20260928 06:45:44.139947 15113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:44.139927 15116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:44.140645 15108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:44.141366 15108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:44.142549 15108 hybrid_clock.cc:648] HybridClock initialized: now 1790577944142514 us; error 55 us; skew 500 ppm
Sep 28 06:45:44 dist-test-slave-twl6 krb5kdc[14851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577944, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:44.145344 15108 init.cc:377] Logged in from keytab as oryx/127.7.48.2@KRBTEST.COM (short username oryx)
I20260928 06:45:44.146307 15108 webserver.cc:460] Webserver started at http://127.7.48.2:33557/ using document root <none> and password file <none>
I20260928 06:45:44.146894 15108 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:44.146989 15108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:44.147233 15108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:44.149420 15108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "bd72f0024ada43e7a14d4e1bee99f7e6"
format_stamp: "Formatted at 2026-09-28 06:45:44 on dist-test-slave-twl6"
server_key: "3c59729b5e7ac334059c762d2822d72f"
server_key_iv: "851052a7af3f18a4ee4cdbaf7f810489"
server_key_version: "encryptionkey@0"
I20260928 06:45:44.150005 15108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "bd72f0024ada43e7a14d4e1bee99f7e6"
format_stamp: "Formatted at 2026-09-28 06:45:44 on dist-test-slave-twl6"
server_key: "3c59729b5e7ac334059c762d2822d72f"
server_key_iv: "851052a7af3f18a4ee4cdbaf7f810489"
server_key_version: "encryptionkey@0"
I20260928 06:45:44.153916 15108 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:45:44.156467 15123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:44.157713 15108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:45:44.157855 15108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "bd72f0024ada43e7a14d4e1bee99f7e6"
format_stamp: "Formatted at 2026-09-28 06:45:44 on dist-test-slave-twl6"
server_key: "3c59729b5e7ac334059c762d2822d72f"
server_key_iv: "851052a7af3f18a4ee4cdbaf7f810489"
server_key_version: "encryptionkey@0"
I20260928 06:45:44.157991 15108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:44.199225 15108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:44.202540 15108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:44.202762 15108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:44.204424 15108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:44.204499 15108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:44.204569 15108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:44.204612 15108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:45:44 dist-test-slave-twl6 krb5kdc[14851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577944, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM
I20260928 06:45:44.216107 15108 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:40247
I20260928 06:45:44.216157 15238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:40247 every 8 connection(s)
I20260928 06:45:44.217069 15108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:45:44.218847 14945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.205159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:54881 (local address 127.7.48.62:46109)
0928 06:45:44.205301 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:44.205304 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:44.206472 (+ 1168us) server_negotiation.cc:407] Connection header received
0928 06:45:44.207503 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:44.207510 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:44.207581 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:44.207665 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:44.209122 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.209772 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.211054 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.211283 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.211404 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.214941 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:44.214945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:44.214948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:44.214979 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:44.216770 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:44.217302 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:44.217306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:44.217308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:44.217400 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:44.217799 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:44.217802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:44.217804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:44.218057 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:44.218166 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:44.218609 (+ 443us) server_negotiation.cc:299] Negotiation successful
0928 06:45:44.218727 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:45:44.219740 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.205763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:46109 (local address 127.7.48.2:54881)
0928 06:45:44.206307 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:44.206359 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:44.207239 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:44.208146 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:44.208154 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:44.208482 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:44.208965 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:44.208977 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.209905 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.209909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:44.210917 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:44.210927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.211459 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.211463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:44.211702 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.212527 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:44.212544 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:44.214777 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:44.216892 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:44.216900 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:44.216915 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:44.217207 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:44.217506 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:44.217510 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:44.217515 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:44.217682 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:44.218168 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:44.218173 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:44.218394 (+ 221us) client_negotiation.cc:769] Sending connection context
0928 06:45:44.218559 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 06:45:44.218839 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":144,"threads_started":1}
I20260928 06:45:44.220960 15239 heartbeater.cc:347] Connected to a master server at 127.7.48.62:46109
I20260928 06:45:44.221102 15239 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:44.221712 15239 heartbeater.cc:511] Master 127.7.48.62:46109 requested a full tablet report, sending...
I20260928 06:45:44.222891 14896 ts_manager.cc:194] Registered new tserver with Master: bd72f0024ada43e7a14d4e1bee99f7e6 (127.7.48.2:40247)
I20260928 06:45:44.223662 14896 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.48.2@KRBTEST.COM'} at 127.7.48.2:54881
I20260928 06:45:44.226724 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 15108
I20260928 06:45:44.226851 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:45:44.227185 7360 external_mini_cluster.cc:1428] Setting key 167358b17450e91e2fb65c070208fd05
I20260928 06:45:44.231019 14945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.223413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:42853 (local address 127.7.48.62:46109)
0928 06:45:44.223528 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:44.223532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:44.224172 (+ 640us) server_negotiation.cc:407] Connection header received
0928 06:45:44.224296 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:44.224299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:44.224348 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:44.224433 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:44.224836 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.225570 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.226337 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.226540 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.226623 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.227830 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:44.227833 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:44.227836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:44.227860 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:44.229316 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:44.229748 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:44.229752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:44.229754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:44.229838 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:44.230120 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:44.230124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:44.230126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:44.230343 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:44.230454 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:44.230753 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 06:45:44.230858 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:45:44.231035 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.223343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:46109 (local address 127.7.48.2:42853)
0928 06:45:44.224083 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:44.224097 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:44.224195 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:44.224454 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:44.224456 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:44.224616 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:44.224723 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:44.224727 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.225696 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.225698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:44.226237 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:44.226244 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.226664 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.226667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:44.226767 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.227257 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:44.227273 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:44.227693 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:44.229434 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:44.229440 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:44.229442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:44.229645 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:44.229937 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:44.229940 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:44.229942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:44.230029 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:44.230444 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:44.230448 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:44.230568 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 06:45:44.230699 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 06:45:44.230850 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for oryx/127.7.48.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.48.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.3.keytab.
Entry for principal oryx/127.7.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.3.keytab.
I20260928 06:45:44.255483 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:46109
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.3.keytab
--principal=oryx/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:44.370855 15248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:44.371163 15248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:44.371243 15248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:44.375325 15248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:44.375427 15248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:44.375505 15248 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:45:44.375586 15248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:45:44.380909 15248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.3.keytab
--principal=oryx/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.48.62:46109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:44.382076 15248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:44.383217 15248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:44.393060 15253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:44.393049 15256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:44.393033 15254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:44.393189 15248 server_base.cc:1034] running on GCE node
I20260928 06:45:44.393705 15248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:44.394356 15248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:44.395548 15248 hybrid_clock.cc:648] HybridClock initialized: now 1790577944395514 us; error 54 us; skew 500 ppm
Sep 28 06:45:44 dist-test-slave-twl6 krb5kdc[14851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577944, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:44.398372 15248 init.cc:377] Logged in from keytab as oryx/127.7.48.3@KRBTEST.COM (short username oryx)
I20260928 06:45:44.399267 15248 webserver.cc:460] Webserver started at http://127.7.48.3:35507/ using document root <none> and password file <none>
I20260928 06:45:44.399856 15248 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:44.400023 15248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:44.400272 15248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:44.402403 15248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "f2778a9828114f36ae7f9421329e9dfd"
format_stamp: "Formatted at 2026-09-28 06:45:44 on dist-test-slave-twl6"
server_key: "b9d08aed7596fdd2fa8e557bf1e3eef6"
server_key_iv: "bfb315fbfdb394baead7dfb27ffca996"
server_key_version: "encryptionkey@0"
I20260928 06:45:44.402995 15248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "f2778a9828114f36ae7f9421329e9dfd"
format_stamp: "Formatted at 2026-09-28 06:45:44 on dist-test-slave-twl6"
server_key: "b9d08aed7596fdd2fa8e557bf1e3eef6"
server_key_iv: "bfb315fbfdb394baead7dfb27ffca996"
server_key_version: "encryptionkey@0"
I20260928 06:45:44.406844 15248 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 06:45:44.409366 15263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:44.410643 15248 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:45:44.410795 15248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "f2778a9828114f36ae7f9421329e9dfd"
format_stamp: "Formatted at 2026-09-28 06:45:44 on dist-test-slave-twl6"
server_key: "b9d08aed7596fdd2fa8e557bf1e3eef6"
server_key_iv: "bfb315fbfdb394baead7dfb27ffca996"
server_key_version: "encryptionkey@0"
I20260928 06:45:44.410909 15248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:44.420817 15248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:44.424173 15248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:44.424377 15248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:44.425849 15248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:44.425928 15248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:44.425997 15248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:44.426047 15248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:45:44 dist-test-slave-twl6 krb5kdc[14851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577944, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM
I20260928 06:45:44.437175 15248 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:36477
I20260928 06:45:44.437428 15378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:36477 every 8 connection(s)
I20260928 06:45:44.438164 15248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:45:44.440642 14945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.426744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:57989 (local address 127.7.48.62:46109)
0928 06:45:44.426917 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:44.426924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:44.427769 (+ 845us) server_negotiation.cc:407] Connection header received
0928 06:45:44.428773 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:44.428776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:44.428837 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:44.428924 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:44.430275 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.431218 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.433244 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.433640 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.433755 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.436720 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:44.436724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:44.436727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:44.436772 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:44.438524 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:44.439179 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:44.439184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:44.439189 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:44.439292 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:44.439608 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:44.439610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:44.439611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:44.439795 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:44.439868 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:44.440408 (+ 540us) server_negotiation.cc:299] Negotiation successful
0928 06:45:44.440515 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 06:45:44.441442 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.427140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:46109 (local address 127.7.48.3:57989)
0928 06:45:44.427597 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:44.427649 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:44.428553 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:44.429279 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:44.429291 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:44.429639 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:44.430113 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:44.430126 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.431399 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.431406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:44.433073 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:44.433087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.433935 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.433939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:44.434085 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.434740 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:44.434765 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:44.436488 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:44.438670 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:44.438675 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:44.438690 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:44.439032 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:44.439383 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:44.439385 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:44.439386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:44.439517 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:44.439933 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:44.439940 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:44.440185 (+ 245us) client_negotiation.cc:769] Sending connection context
0928 06:45:44.440340 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 06:45:44.440591 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":142,"threads_started":1}
I20260928 06:45:44.442013 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 15248
I20260928 06:45:44.442179 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:45:44.442493 7360 external_mini_cluster.cc:1428] Setting key 93faa0c75fbcd7f8d0a47f51dbc9c4dc
I20260928 06:45:44.442755 15379 heartbeater.cc:347] Connected to a master server at 127.7.48.62:46109
I20260928 06:45:44.442936 15379 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:44.443563 15379 heartbeater.cc:511] Master 127.7.48.62:46109 requested a full tablet report, sending...
I20260928 06:45:44.444802 14895 ts_manager.cc:194] Registered new tserver with Master: f2778a9828114f36ae7f9421329e9dfd (127.7.48.3:36477)
I20260928 06:45:44.445480 14895 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.48.3@KRBTEST.COM'} at 127.7.48.3:57989
I20260928 06:45:44.452106 14945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.445755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:41913 (local address 127.7.48.62:46109)
0928 06:45:44.445892 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:44.445896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:44.445907 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:45:44.445938 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:44.445940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:44.445975 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:44.446062 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:44.446506 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.447019 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.447845 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.448110 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.448225 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.449279 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:44.449282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:44.449285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:44.449309 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:44.450452 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:44.450877 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:44.450881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:44.450883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:44.450964 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:44.451249 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:44.451254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:44.451256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:44.451431 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:44.451534 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:44.451800 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 06:45:44.451947 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:45:44.452122 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.445091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:46109 (local address 127.7.48.3:41913)
0928 06:45:44.445350 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:44.445364 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:44.445437 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:44.446088 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:44.446091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:44.446288 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:44.446386 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:44.446392 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.447154 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.447157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:44.447719 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:44.447726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.448227 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.448231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:44.448334 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.448797 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:44.448808 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:44.449141 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:44.450568 (+ 1427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:44.450572 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:44.450574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:44.450772 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:44.451057 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:44.451061 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:44.451063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:44.451154 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:44.451536 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:44.451540 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:44.451633 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 06:45:44.451744 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 06:45:44.451889 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":96,"threads_started":1}
I20260928 06:45:44.456291 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:45:44 dist-test-slave-twl6 krb5kdc[14851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577944, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:45:44 dist-test-slave-twl6 krb5kdc[14851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:44 dist-test-slave-twl6 krb5kdc[14851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:45:44.481280 14945 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:44.474544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56478 (local address 127.7.48.62:46109)
0928 06:45:44.474702 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:44.474705 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:44.474718 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:45:44.474762 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:44.474764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:44.474804 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:44.474904 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:44.475630 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.476232 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.477112 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.477351 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.477469 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.481099 (+ 3630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56478: BlockingRecv error: recv got EOF from 127.0.0.1:56478 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 06:45:44 dist-test-slave-twl6 krb5kdc[14851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577944, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM
I20260928 06:45:44.491112 14945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.482744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56494 (local address 127.7.48.62:46109)
0928 06:45:44.482883 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:44.482888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:44.483034 (+ 146us) server_negotiation.cc:407] Connection header received
0928 06:45:44.483188 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:44.483191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:44.483239 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:44.483324 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:44.483994 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.484575 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.485425 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.485627 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.485715 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.487997 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:44.488000 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:44.488002 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:44.488027 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:44.489556 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:44.489972 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:44.489975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:44.489976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:44.490049 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:44.490281 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:44.490283 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:44.490284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:44.490445 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:44.490545 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:44.490799 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 06:45:44.490889 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 06:45:44.494035 14895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:45:44.497041 14895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:45:44.511459 15404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.505896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:40247 (local address 127.0.0.1:39484)
0928 06:45:44.506512 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:44.506526 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:44.506616 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:44.507279 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:44.507281 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:44.507292 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:44.507407 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:44.507411 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.509055 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.509058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:44.510110 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:44.510120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.511062 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.511064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:44.511146 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.511178 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 06:45:44.511256 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:45:44.511339 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":76,"thread_start_us":93,"threads_started":1}
I20260928 06:45:44.511557 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.506330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39484 (local address 127.7.48.2:40247)
0928 06:45:44.506865 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:44.506876 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:44.506894 (+ 18us) server_negotiation.cc:407] Connection header received
0928 06:45:44.506958 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:44.506962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:44.507129 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:44.507295 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:44.507525 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.508778 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.510242 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.510942 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.511052 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.511224 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:44.511298 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 06:45:44.511391 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":103,"threads_started":1}
I20260928 06:45:44.512028 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.506231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:36379 (local address 127.0.0.1:60448)
0928 06:45:44.507037 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:44.507047 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:44.507164 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:44.507912 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:44.507914 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:44.507932 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:44.508016 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:44.508022 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.509111 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.509116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:44.510515 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:44.510521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.511564 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.511568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:44.511681 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.511716 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 06:45:44.511797 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 06:45:44.511859 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":86,"thread_start_us":89,"threads_started":1}
I20260928 06:45:44.512356 15406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.506694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60448 (local address 127.7.48.1:36379)
0928 06:45:44.506929 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:44.506937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:44.507531 (+ 594us) server_negotiation.cc:407] Connection header received
0928 06:45:44.507603 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:44.507606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:44.507762 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:44.507948 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:44.508112 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.508894 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.510645 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.511433 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.511539 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.511698 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:44.511853 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 06:45:44.511975 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":77,"threads_started":1}
I20260928 06:45:44.513123 15401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.505440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:36477 (local address 127.0.0.1:59870)
0928 06:45:44.505859 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:44.505916 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:44.506062 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:44.507178 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:44.507187 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:44.507214 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:44.507403 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:44.507411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.508952 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.508959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:44.510509 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:44.510521 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.512722 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.512728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:44.512813 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.512844 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 06:45:44.512931 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:45:44.512998 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":200,"thread_start_us":97,"threads_started":1}
I20260928 06:45:44.513192 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.505605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59870 (local address 127.7.48.3:36477)
0928 06:45:44.506627 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:44.506642 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:44.506668 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:45:44.506772 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:44.506778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:44.507005 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:44.507198 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:44.507525 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.508777 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.511759 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.512583 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.512705 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.512862 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:44.512956 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 06:45:44.513041 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":888,"thread_start_us":125,"threads_started":1}
I20260928 06:45:44.515048 15173 tablet_service.cc:1467] Processing CreateTablet for tablet 00bc7940303b423dbff515526289d15f (DEFAULT_TABLE table=test-table [id=399ac3261bfc45c4a6c231d6e54a59e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:44.515734 15313 tablet_service.cc:1467] Processing CreateTablet for tablet 00bc7940303b423dbff515526289d15f (DEFAULT_TABLE table=test-table [id=399ac3261bfc45c4a6c231d6e54a59e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:44.516355 15173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00bc7940303b423dbff515526289d15f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:44.516742 15313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00bc7940303b423dbff515526289d15f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:44.522054 15033 tablet_service.cc:1467] Processing CreateTablet for tablet 00bc7940303b423dbff515526289d15f (DEFAULT_TABLE table=test-table [id=399ac3261bfc45c4a6c231d6e54a59e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:44.523188 15033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00bc7940303b423dbff515526289d15f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:44.524935 15407 tablet_bootstrap.cc:492] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd: Bootstrap starting.
I20260928 06:45:44.526242 15408 tablet_bootstrap.cc:492] T 00bc7940303b423dbff515526289d15f P bd72f0024ada43e7a14d4e1bee99f7e6: Bootstrap starting.
I20260928 06:45:44.527266 15407 tablet_bootstrap.cc:654] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:44.528154 15407 log.cc:824] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:44.528380 15408 tablet_bootstrap.cc:654] T 00bc7940303b423dbff515526289d15f P bd72f0024ada43e7a14d4e1bee99f7e6: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:44.529250 15408 log.cc:824] T 00bc7940303b423dbff515526289d15f P bd72f0024ada43e7a14d4e1bee99f7e6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:44.529947 15411 tablet_bootstrap.cc:492] T 00bc7940303b423dbff515526289d15f P 69f4f371eac34d12bca22a7bda991178: Bootstrap starting.
I20260928 06:45:44.530469 15407 tablet_bootstrap.cc:492] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd: No bootstrap required, opened a new log
I20260928 06:45:44.530681 15407 ts_tablet_manager.cc:1397] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 06:45:44.531198 15408 tablet_bootstrap.cc:492] T 00bc7940303b423dbff515526289d15f P bd72f0024ada43e7a14d4e1bee99f7e6: No bootstrap required, opened a new log
I20260928 06:45:44.531383 15408 ts_tablet_manager.cc:1397] T 00bc7940303b423dbff515526289d15f P bd72f0024ada43e7a14d4e1bee99f7e6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 06:45:44.532162 15411 tablet_bootstrap.cc:654] T 00bc7940303b423dbff515526289d15f P 69f4f371eac34d12bca22a7bda991178: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:44.532964 15411 log.cc:824] T 00bc7940303b423dbff515526289d15f P 69f4f371eac34d12bca22a7bda991178: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:44.534273 15407 raft_consensus.cc:348] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } } peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } }
I20260928 06:45:44.534624 15407 raft_consensus.cc:374] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:44.534698 15407 raft_consensus.cc:729] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2778a9828114f36ae7f9421329e9dfd, State: Initialized, Role: FOLLOWER
I20260928 06:45:44.534873 15411 tablet_bootstrap.cc:492] T 00bc7940303b423dbff515526289d15f P 69f4f371eac34d12bca22a7bda991178: No bootstrap required, opened a new log
I20260928 06:45:44.535048 15411 ts_tablet_manager.cc:1397] T 00bc7940303b423dbff515526289d15f P 69f4f371eac34d12bca22a7bda991178: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 06:45:44.535161 15407 consensus_queue.cc:260] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } } peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } }
I20260928 06:45:44.535313 15408 raft_consensus.cc:348] T 00bc7940303b423dbff515526289d15f P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } } peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } }
I20260928 06:45:44.535548 15408 raft_consensus.cc:374] T 00bc7940303b423dbff515526289d15f P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:44.535602 15408 raft_consensus.cc:729] T 00bc7940303b423dbff515526289d15f P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd72f0024ada43e7a14d4e1bee99f7e6, State: Initialized, Role: FOLLOWER
I20260928 06:45:44.535827 15379 heartbeater.cc:503] Master 127.7.48.62:46109 was elected leader, sending a full tablet report...
I20260928 06:45:44.536177 15407 ts_tablet_manager.cc:1428] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 06:45:44.536281 15408 consensus_queue.cc:260] T 00bc7940303b423dbff515526289d15f P bd72f0024ada43e7a14d4e1bee99f7e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } } peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } }
I20260928 06:45:44.537478 15408 ts_tablet_manager.cc:1428] T 00bc7940303b423dbff515526289d15f P bd72f0024ada43e7a14d4e1bee99f7e6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 06:45:44.537810 15239 heartbeater.cc:503] Master 127.7.48.62:46109 was elected leader, sending a full tablet report...
I20260928 06:45:44.539541 15411 raft_consensus.cc:348] T 00bc7940303b423dbff515526289d15f P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } } peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } }
I20260928 06:45:44.539799 15411 raft_consensus.cc:374] T 00bc7940303b423dbff515526289d15f P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:44.539850 15411 raft_consensus.cc:729] T 00bc7940303b423dbff515526289d15f P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f4f371eac34d12bca22a7bda991178, State: Initialized, Role: FOLLOWER
I20260928 06:45:44.540585 15411 consensus_queue.cc:260] T 00bc7940303b423dbff515526289d15f P 69f4f371eac34d12bca22a7bda991178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } } peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } }
I20260928 06:45:44.541245 15099 heartbeater.cc:503] Master 127.7.48.62:46109 was elected leader, sending a full tablet report...
I20260928 06:45:44.541497 15411 ts_tablet_manager.cc:1428] T 00bc7940303b423dbff515526289d15f P 69f4f371eac34d12bca22a7bda991178: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260928 06:45:44.608475 15413 raft_consensus.cc:482] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:45:44.608729 15413 raft_consensus.cc:504] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } } peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } }
I20260928 06:45:44.609920 15413 leader_election.cc:288] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69f4f371eac34d12bca22a7bda991178 (127.7.48.1:36379), bd72f0024ada43e7a14d4e1bee99f7e6 (127.7.48.2:40247)
I20260928 06:45:44.614224 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.610372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:36291 (local address 127.7.48.2:40247)
0928 06:45:44.610509 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:44.610515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:44.610596 (+ 81us) server_negotiation.cc:407] Connection header received
0928 06:45:44.610783 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:44.610792 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:44.610838 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:44.610928 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:44.611203 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.611726 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.613025 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.613590 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.613708 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.613776 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:44.614015 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 06:45:44.614095 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 06:45:44.614239 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.610257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:40247 (local address 127.7.48.3:36291)
0928 06:45:44.610496 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:44.610513 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:44.610648 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:44.610942 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:44.610945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:44.610963 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:44.611086 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:44.611091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.611849 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.611854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:44.612870 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:44.612881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.613721 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.613725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:44.613842 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.613905 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 06:45:44.613999 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 06:45:44.614088 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260928 06:45:44.614627 15406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.610494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:49261 (local address 127.7.48.1:36379)
0928 06:45:44.610618 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:44.610624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:44.610846 (+ 222us) server_negotiation.cc:407] Connection header received
0928 06:45:44.611001 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:44.611004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:44.611056 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:44.611141 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:44.611760 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.612289 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.613581 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.614005 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.614129 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.614189 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:44.614433 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 06:45:44.614513 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 06:45:44.614976 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.610258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:36379 (local address 127.7.48.3:49261)
0928 06:45:44.610737 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:44.610748 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:44.610886 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:44.611159 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:44.611162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:44.611177 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:44.611638 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:44.611645 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.612403 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.612407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:44.613384 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:44.613394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.614143 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.614147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:44.614250 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.614272 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 06:45:44.614770 (+ 498us) client_negotiation.cc:240] Negotiation successful
0928 06:45:44.614849 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":42,"thread_start_us":144,"threads_started":1}
I20260928 06:45:44.614949 15193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00bc7940303b423dbff515526289d15f" candidate_uuid: "f2778a9828114f36ae7f9421329e9dfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" is_pre_election: true
I20260928 06:45:44.615339 15193 raft_consensus.cc:2454] T 00bc7940303b423dbff515526289d15f P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2778a9828114f36ae7f9421329e9dfd in term 0.
I20260928 06:45:44.615782 15053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00bc7940303b423dbff515526289d15f" candidate_uuid: "f2778a9828114f36ae7f9421329e9dfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f4f371eac34d12bca22a7bda991178" is_pre_election: true
I20260928 06:45:44.615864 15264 leader_election.cc:302] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd72f0024ada43e7a14d4e1bee99f7e6, f2778a9828114f36ae7f9421329e9dfd; no voters:
I20260928 06:45:44.616115 15053 raft_consensus.cc:2454] T 00bc7940303b423dbff515526289d15f P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2778a9828114f36ae7f9421329e9dfd in term 0.
I20260928 06:45:44.616233 15413 raft_consensus.cc:2781] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:45:44.616360 15413 raft_consensus.cc:482] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:45:44.616447 15413 raft_consensus.cc:3037] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:44.617539 15413 raft_consensus.cc:504] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } } peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } }
I20260928 06:45:44.617916 15413 leader_election.cc:288] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd [CANDIDATE]: Term 1 election: Requested vote from peers 69f4f371eac34d12bca22a7bda991178 (127.7.48.1:36379), bd72f0024ada43e7a14d4e1bee99f7e6 (127.7.48.2:40247)
I20260928 06:45:44.618305 15053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00bc7940303b423dbff515526289d15f" candidate_uuid: "f2778a9828114f36ae7f9421329e9dfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f4f371eac34d12bca22a7bda991178"
I20260928 06:45:44.618304 15193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00bc7940303b423dbff515526289d15f" candidate_uuid: "f2778a9828114f36ae7f9421329e9dfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6"
I20260928 06:45:44.618432 15053 raft_consensus.cc:3037] T 00bc7940303b423dbff515526289d15f P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:44.618432 15193 raft_consensus.cc:3037] T 00bc7940303b423dbff515526289d15f P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:44.619761 15193 raft_consensus.cc:2454] T 00bc7940303b423dbff515526289d15f P bd72f0024ada43e7a14d4e1bee99f7e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2778a9828114f36ae7f9421329e9dfd in term 1.
I20260928 06:45:44.619761 15053 raft_consensus.cc:2454] T 00bc7940303b423dbff515526289d15f P 69f4f371eac34d12bca22a7bda991178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2778a9828114f36ae7f9421329e9dfd in term 1.
I20260928 06:45:44.620121 15264 leader_election.cc:302] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 69f4f371eac34d12bca22a7bda991178, bd72f0024ada43e7a14d4e1bee99f7e6, f2778a9828114f36ae7f9421329e9dfd; no voters:
I20260928 06:45:44.620277 15413 raft_consensus.cc:2781] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:44.620488 15413 raft_consensus.cc:686] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd [term 1 LEADER]: Becoming Leader. State: Replica: f2778a9828114f36ae7f9421329e9dfd, State: Running, Role: LEADER
I20260928 06:45:44.620884 15413 consensus_queue.cc:237] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } } peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } }
I20260928 06:45:44.624370 14895 catalog_manager.cc:5495] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd reported cstate change: term changed from 0 to 1, leader changed from <none> to f2778a9828114f36ae7f9421329e9dfd (127.7.48.3). New cstate: current_term: 1 leader_uuid: "f2778a9828114f36ae7f9421329e9dfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } health_report { overall_health: UNKNOWN } } }
I20260928 06:45:44.645912 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.642175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35514 (local address 127.7.48.3:36477)
0928 06:45:44.642349 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:44.642356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:44.642371 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:45:44.642453 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:44.642455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:44.642511 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:44.642640 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:45:44.642896 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.643407 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.644377 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.644599 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.644712 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.645008 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:45:44.645472 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:45:44.645607 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:44.645679 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 06:45:44.645740 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 06:45:44.652146 15053 raft_consensus.cc:1261] T 00bc7940303b423dbff515526289d15f P 69f4f371eac34d12bca22a7bda991178 [term 1 FOLLOWER]: Refusing update from remote peer f2778a9828114f36ae7f9421329e9dfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:44.652379 15193 raft_consensus.cc:1261] T 00bc7940303b423dbff515526289d15f P bd72f0024ada43e7a14d4e1bee99f7e6 [term 1 FOLLOWER]: Refusing update from remote peer f2778a9828114f36ae7f9421329e9dfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:44.652904 15413 consensus_queue.cc:1035] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:44.653124 15417 consensus_queue.cc:1035] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:44.660542 15420 mvcc.cc:205] Tried to move back new op lower bound from 7334207261287305216 to 7334207261170012160. Current Snapshot: MvccSnapshot[applied={T|T < 7334207261287305216}]
I20260928 06:45:44.662618 15421 mvcc.cc:205] Tried to move back new op lower bound from 7334207261287305216 to 7334207261170012160. Current Snapshot: MvccSnapshot[applied={T|T < 7334207261287305216}]
I20260928 06:45:44.669775 14895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:44.669961 14895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:44.672358 14895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc: Sending DeleteTablet for 3 replicas of tablet 00bc7940303b423dbff515526289d15f
I20260928 06:45:44.673169 15313 tablet_service.cc:1514] Processing DeleteTablet for tablet 00bc7940303b423dbff515526289d15f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:44 UTC) from {username='oryx', principal='oryx/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:59870
I20260928 06:45:44.673259 15173 tablet_service.cc:1514] Processing DeleteTablet for tablet 00bc7940303b423dbff515526289d15f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:44 UTC) from {username='oryx', principal='oryx/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:39484
I20260928 06:45:44.674041 15033 tablet_service.cc:1514] Processing DeleteTablet for tablet 00bc7940303b423dbff515526289d15f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:44 UTC) from {username='oryx', principal='oryx/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:60448
I20260928 06:45:44.674479 15430 tablet_replica.cc:330] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd: stopping tablet replica
I20260928 06:45:44.674460 14895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 06:45:44.674785 15430 raft_consensus.cc:2229] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd [term 1 LEADER]: Raft consensus shutting down.
W20260928 06:45:44.674880 14895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:45:44.675173 15430 raft_consensus.cc:2258] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:44.677980 15431 tablet_replica.cc:330] T 00bc7940303b423dbff515526289d15f P 69f4f371eac34d12bca22a7bda991178: stopping tablet replica
I20260928 06:45:44.678323 15431 raft_consensus.cc:2229] T 00bc7940303b423dbff515526289d15f P 69f4f371eac34d12bca22a7bda991178 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:44.678610 15429 tablet_replica.cc:330] T 00bc7940303b423dbff515526289d15f P bd72f0024ada43e7a14d4e1bee99f7e6: stopping tablet replica
I20260928 06:45:44.678638 15431 raft_consensus.cc:2258] T 00bc7940303b423dbff515526289d15f P 69f4f371eac34d12bca22a7bda991178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:44.678893 15429 raft_consensus.cc:2229] T 00bc7940303b423dbff515526289d15f P bd72f0024ada43e7a14d4e1bee99f7e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:44.680244 15033 tablet_service.cc:1467] Processing CreateTablet for tablet 32afe850044947f78fd909ac6177389e (DEFAULT_TABLE table=test-table [id=f41fb61db2a74b8ca9ae898e174c929b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:44.680533 15033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32afe850044947f78fd909ac6177389e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:44.680817 15173 tablet_service.cc:1467] Processing CreateTablet for tablet 32afe850044947f78fd909ac6177389e (DEFAULT_TABLE table=test-table [id=f41fb61db2a74b8ca9ae898e174c929b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:44.681108 15173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32afe850044947f78fd909ac6177389e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:44.681063 15313 tablet_service.cc:1467] Processing CreateTablet for tablet 32afe850044947f78fd909ac6177389e (DEFAULT_TABLE table=test-table [id=f41fb61db2a74b8ca9ae898e174c929b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:44.681327 15313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32afe850044947f78fd909ac6177389e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:44.683179 15411 tablet_bootstrap.cc:492] T 32afe850044947f78fd909ac6177389e P 69f4f371eac34d12bca22a7bda991178: Bootstrap starting.
I20260928 06:45:44.683789 15407 tablet_bootstrap.cc:492] T 32afe850044947f78fd909ac6177389e P f2778a9828114f36ae7f9421329e9dfd: Bootstrap starting.
I20260928 06:45:44.683854 15408 tablet_bootstrap.cc:492] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6: Bootstrap starting.
I20260928 06:45:44.684207 15411 tablet_bootstrap.cc:654] T 32afe850044947f78fd909ac6177389e P 69f4f371eac34d12bca22a7bda991178: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:44.684768 15430 ts_tablet_manager.cc:1905] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:44.684837 15407 tablet_bootstrap.cc:654] T 32afe850044947f78fd909ac6177389e P f2778a9828114f36ae7f9421329e9dfd: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:44.684882 15408 tablet_bootstrap.cc:654] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:44.685276 15431 ts_tablet_manager.cc:1905] T 00bc7940303b423dbff515526289d15f P 69f4f371eac34d12bca22a7bda991178: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:44.685989 15429 raft_consensus.cc:2258] T 00bc7940303b423dbff515526289d15f P bd72f0024ada43e7a14d4e1bee99f7e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:44.687820 15431 ts_tablet_manager.cc:1918] T 00bc7940303b423dbff515526289d15f P 69f4f371eac34d12bca22a7bda991178: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:45:44.687937 15431 log.cc:1196] T 00bc7940303b423dbff515526289d15f P 69f4f371eac34d12bca22a7bda991178: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-0/wal/wals/00bc7940303b423dbff515526289d15f
I20260928 06:45:44.688264 15431 ts_tablet_manager.cc:1939] T 00bc7940303b423dbff515526289d15f P 69f4f371eac34d12bca22a7bda991178: Deleting consensus metadata
I20260928 06:45:44.689555 14882 catalog_manager.cc:4841] TS 69f4f371eac34d12bca22a7bda991178 (127.7.48.1:36379): tablet 00bc7940303b423dbff515526289d15f (table test-table [id=399ac3261bfc45c4a6c231d6e54a59e8]) successfully deleted
I20260928 06:45:44.689406 15429 ts_tablet_manager.cc:1905] T 00bc7940303b423dbff515526289d15f P bd72f0024ada43e7a14d4e1bee99f7e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:44.690397 15411 tablet_bootstrap.cc:492] T 32afe850044947f78fd909ac6177389e P 69f4f371eac34d12bca22a7bda991178: No bootstrap required, opened a new log
I20260928 06:45:44.690478 15411 ts_tablet_manager.cc:1397] T 32afe850044947f78fd909ac6177389e P 69f4f371eac34d12bca22a7bda991178: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260928 06:45:44.691231 15411 raft_consensus.cc:348] T 32afe850044947f78fd909ac6177389e P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:44.691337 15411 raft_consensus.cc:374] T 32afe850044947f78fd909ac6177389e P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:44.691366 15411 raft_consensus.cc:729] T 32afe850044947f78fd909ac6177389e P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f4f371eac34d12bca22a7bda991178, State: Initialized, Role: FOLLOWER
I20260928 06:45:44.691481 15411 consensus_queue.cc:260] T 32afe850044947f78fd909ac6177389e P 69f4f371eac34d12bca22a7bda991178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:44.691736 15411 ts_tablet_manager.cc:1428] T 32afe850044947f78fd909ac6177389e P 69f4f371eac34d12bca22a7bda991178: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 06:45:44.692229 15430 ts_tablet_manager.cc:1918] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:45:44.692384 15430 log.cc:1196] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/00bc7940303b423dbff515526289d15f
I20260928 06:45:44.693126 15429 ts_tablet_manager.cc:1918] T 00bc7940303b423dbff515526289d15f P bd72f0024ada43e7a14d4e1bee99f7e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:45:44.693250 15429 log.cc:1196] T 00bc7940303b423dbff515526289d15f P bd72f0024ada43e7a14d4e1bee99f7e6: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/00bc7940303b423dbff515526289d15f
I20260928 06:45:44.693478 15430 ts_tablet_manager.cc:1939] T 00bc7940303b423dbff515526289d15f P f2778a9828114f36ae7f9421329e9dfd: Deleting consensus metadata
I20260928 06:45:44.693576 15429 ts_tablet_manager.cc:1939] T 00bc7940303b423dbff515526289d15f P bd72f0024ada43e7a14d4e1bee99f7e6: Deleting consensus metadata
I20260928 06:45:44.694779 14880 catalog_manager.cc:4841] TS bd72f0024ada43e7a14d4e1bee99f7e6 (127.7.48.2:40247): tablet 00bc7940303b423dbff515526289d15f (table test-table [id=399ac3261bfc45c4a6c231d6e54a59e8]) successfully deleted
I20260928 06:45:44.694779 14882 catalog_manager.cc:4841] TS f2778a9828114f36ae7f9421329e9dfd (127.7.48.3:36477): tablet 00bc7940303b423dbff515526289d15f (table test-table [id=399ac3261bfc45c4a6c231d6e54a59e8]) successfully deleted
I20260928 06:45:44.696720 15407 tablet_bootstrap.cc:492] T 32afe850044947f78fd909ac6177389e P f2778a9828114f36ae7f9421329e9dfd: No bootstrap required, opened a new log
I20260928 06:45:44.696720 15408 tablet_bootstrap.cc:492] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6: No bootstrap required, opened a new log
I20260928 06:45:44.696812 15408 ts_tablet_manager.cc:1397] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20260928 06:45:44.696812 15407 ts_tablet_manager.cc:1397] T 32afe850044947f78fd909ac6177389e P f2778a9828114f36ae7f9421329e9dfd: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260928 06:45:44.697290 15407 raft_consensus.cc:348] T 32afe850044947f78fd909ac6177389e P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:44.697290 15408 raft_consensus.cc:348] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:44.697403 15407 raft_consensus.cc:374] T 32afe850044947f78fd909ac6177389e P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:44.697403 15408 raft_consensus.cc:374] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:44.697438 15408 raft_consensus.cc:729] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd72f0024ada43e7a14d4e1bee99f7e6, State: Initialized, Role: FOLLOWER
I20260928 06:45:44.697438 15407 raft_consensus.cc:729] T 32afe850044947f78fd909ac6177389e P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2778a9828114f36ae7f9421329e9dfd, State: Initialized, Role: FOLLOWER
I20260928 06:45:44.697562 15407 consensus_queue.cc:260] T 32afe850044947f78fd909ac6177389e P f2778a9828114f36ae7f9421329e9dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:44.697562 15408 consensus_queue.cc:260] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:44.697815 15408 ts_tablet_manager.cc:1428] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:45:44.697815 15407 ts_tablet_manager.cc:1428] T 32afe850044947f78fd909ac6177389e P f2778a9828114f36ae7f9421329e9dfd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260928 06:45:44.714917 15100 tablet.cc:2369] T 32afe850044947f78fd909ac6177389e P 69f4f371eac34d12bca22a7bda991178: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:45:44.718263 15240 tablet.cc:2369] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:45:44.897953 15414 raft_consensus.cc:482] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:45:44.898154 15414 raft_consensus.cc:504] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:44.899323 15414 leader_election.cc:288] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69f4f371eac34d12bca22a7bda991178 (127.7.48.1:36379), f2778a9828114f36ae7f9421329e9dfd (127.7.48.3:36477)
I20260928 06:45:44.903440 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.900209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:51641 (local address 127.7.48.3:36477)
0928 06:45:44.900349 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:44.900354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:44.900367 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:45:44.900526 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:44.900529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:44.900576 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:44.900657 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:44.900903 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.901414 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.902461 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.902880 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.902999 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.903062 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:44.903245 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 06:45:44.903314 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:45:44.903455 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.899588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:36379 (local address 127.7.48.2:53925)
0928 06:45:44.899954 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:44.899969 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:44.900128 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:44.900501 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:44.900503 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:44.900517 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:44.900635 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:44.900640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.901418 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.901423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:44.902288 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:44.902297 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.902972 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.902975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:44.903070 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.903096 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 06:45:44.903187 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 06:45:44.903266 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":118,"threads_started":1}
I20260928 06:45:44.903563 15406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.899673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:53925 (local address 127.7.48.1:36379)
0928 06:45:44.899836 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:44.899841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:44.900085 (+ 244us) server_negotiation.cc:407] Connection header received
0928 06:45:44.900280 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:44.900284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:44.900332 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:44.900380 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:44.900766 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.901295 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.902420 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.902835 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.902957 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.903023 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:44.903204 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 06:45:44.903408 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 06:45:44.903460 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.900037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:36477 (local address 127.7.48.2:51641)
0928 06:45:44.900289 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:44.900303 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:44.900397 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:44.900668 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:44.900671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:44.900687 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:44.900798 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:44.900803 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.901521 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.901524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:44.902332 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:44.902341 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.903001 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.903005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:44.903112 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.903138 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 06:45:44.903226 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:45:44.903297 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":63,"threads_started":1}
I20260928 06:45:44.904022 15052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32afe850044947f78fd909ac6177389e" candidate_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f4f371eac34d12bca22a7bda991178" is_pre_election: true
I20260928 06:45:44.904196 15052 raft_consensus.cc:2454] T 32afe850044947f78fd909ac6177389e P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd72f0024ada43e7a14d4e1bee99f7e6 in term 0.
I20260928 06:45:44.904511 15333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32afe850044947f78fd909ac6177389e" candidate_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2778a9828114f36ae7f9421329e9dfd" is_pre_election: true
I20260928 06:45:44.904603 15126 leader_election.cc:302] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69f4f371eac34d12bca22a7bda991178, bd72f0024ada43e7a14d4e1bee99f7e6; no voters:
I20260928 06:45:44.904781 15333 raft_consensus.cc:2454] T 32afe850044947f78fd909ac6177389e P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd72f0024ada43e7a14d4e1bee99f7e6 in term 0.
I20260928 06:45:44.904910 15414 raft_consensus.cc:2781] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:45:44.904976 15414 raft_consensus.cc:482] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:45:44.905028 15414 raft_consensus.cc:3037] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:44.906095 15414 raft_consensus.cc:504] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:44.906477 15414 leader_election.cc:288] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6 [CANDIDATE]: Term 1 election: Requested vote from peers 69f4f371eac34d12bca22a7bda991178 (127.7.48.1:36379), f2778a9828114f36ae7f9421329e9dfd (127.7.48.3:36477)
I20260928 06:45:44.906838 15052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32afe850044947f78fd909ac6177389e" candidate_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f4f371eac34d12bca22a7bda991178"
I20260928 06:45:44.906881 15333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32afe850044947f78fd909ac6177389e" candidate_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2778a9828114f36ae7f9421329e9dfd"
I20260928 06:45:44.906977 15052 raft_consensus.cc:3037] T 32afe850044947f78fd909ac6177389e P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:44.907009 15333 raft_consensus.cc:3037] T 32afe850044947f78fd909ac6177389e P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:44.908054 15052 raft_consensus.cc:2454] T 32afe850044947f78fd909ac6177389e P 69f4f371eac34d12bca22a7bda991178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd72f0024ada43e7a14d4e1bee99f7e6 in term 1.
I20260928 06:45:44.908108 15333 raft_consensus.cc:2454] T 32afe850044947f78fd909ac6177389e P f2778a9828114f36ae7f9421329e9dfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd72f0024ada43e7a14d4e1bee99f7e6 in term 1.
I20260928 06:45:44.908337 15126 leader_election.cc:302] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69f4f371eac34d12bca22a7bda991178, bd72f0024ada43e7a14d4e1bee99f7e6; no voters:
I20260928 06:45:44.908545 15414 raft_consensus.cc:2781] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:44.908751 15414 raft_consensus.cc:686] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6 [term 1 LEADER]: Becoming Leader. State: Replica: bd72f0024ada43e7a14d4e1bee99f7e6, State: Running, Role: LEADER
I20260928 06:45:44.909053 15414 consensus_queue.cc:237] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:44.911334 14896 catalog_manager.cc:5495] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd72f0024ada43e7a14d4e1bee99f7e6 (127.7.48.2). New cstate: current_term: 1 leader_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } health_report { overall_health: UNKNOWN } } }
W20260928 06:45:44.939390 15380 tablet.cc:2369] T 32afe850044947f78fd909ac6177389e P f2778a9828114f36ae7f9421329e9dfd: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 06:45:44 dist-test-slave-twl6 krb5kdc[14851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577943, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.62@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM
I20260928 06:45:44.958055 15444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.949665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:46109 (local address 127.0.0.1:59454)
0928 06:45:44.949962 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:44.949980 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:44.950104 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:44.950333 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:44.950335 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:44.950752 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:44.950857 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:44.950863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.951569 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.951571 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:44.952196 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:44.952205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.952629 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.952632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:44.952701 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.953259 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:44.953271 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:44.954716 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:44.956326 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:44.956336 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:44.956343 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:44.956568 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:44.956870 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:44.956873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:44.956875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:44.956954 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:44.957361 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:44.957370 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:44.957481 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 06:45:44.957618 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 06:45:44.957821 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":4,"thread_start_us":131,"threads_started":1}
I20260928 06:45:44.958055 14945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:44.949751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59454 (local address 127.7.48.62:46109)
0928 06:45:44.949886 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:44.949889 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:44.950070 (+ 181us) server_negotiation.cc:407] Connection header received
0928 06:45:44.950175 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:44.950177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:44.950217 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:44.950296 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:44.950965 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.951478 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.952313 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:44.952517 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:44.952594 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:44.954860 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:44.954864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:44.954866 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:44.954885 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:44.956214 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:44.956689 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:44.956693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:44.956695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:44.956769 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:44.957049 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:44.957053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:44.957054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:44.957243 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:44.957349 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:44.957662 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 06:45:44.957875 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":53}
I20260928 06:45:44.962091 14896 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:59454:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 06:45:44.962893 14896 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:45:44.967828 15033 tablet_service.cc:1467] Processing CreateTablet for tablet 4c4a051b514e4f7ea5e108d90db24056 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=82e37de205d54716bef80a371213056e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:45:44.968124 15173 tablet_service.cc:1467] Processing CreateTablet for tablet 4c4a051b514e4f7ea5e108d90db24056 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=82e37de205d54716bef80a371213056e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:45:44.968277 15033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c4a051b514e4f7ea5e108d90db24056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:44.968124 15313 tablet_service.cc:1467] Processing CreateTablet for tablet 4c4a051b514e4f7ea5e108d90db24056 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=82e37de205d54716bef80a371213056e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:45:44.968505 15313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c4a051b514e4f7ea5e108d90db24056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:44.968505 15173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c4a051b514e4f7ea5e108d90db24056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:44.971139 15411 tablet_bootstrap.cc:492] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178: Bootstrap starting.
I20260928 06:45:44.971302 15407 tablet_bootstrap.cc:492] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd: Bootstrap starting.
I20260928 06:45:44.971669 15408 tablet_bootstrap.cc:492] T 4c4a051b514e4f7ea5e108d90db24056 P bd72f0024ada43e7a14d4e1bee99f7e6: Bootstrap starting.
I20260928 06:45:44.971987 15411 tablet_bootstrap.cc:654] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:44.972259 15407 tablet_bootstrap.cc:654] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:44.972647 15408 tablet_bootstrap.cc:654] T 4c4a051b514e4f7ea5e108d90db24056 P bd72f0024ada43e7a14d4e1bee99f7e6: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:44.973466 15411 tablet_bootstrap.cc:492] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178: No bootstrap required, opened a new log
I20260928 06:45:44.973543 15411 ts_tablet_manager.cc:1397] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:45:44.973932 15411 raft_consensus.cc:348] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:44.974044 15411 raft_consensus.cc:374] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:44.974083 15411 raft_consensus.cc:729] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f4f371eac34d12bca22a7bda991178, State: Initialized, Role: FOLLOWER
I20260928 06:45:44.974189 15411 consensus_queue.cc:260] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:44.974365 15407 tablet_bootstrap.cc:492] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd: No bootstrap required, opened a new log
I20260928 06:45:44.974442 15411 ts_tablet_manager.cc:1428] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:45:44.974457 15407 ts_tablet_manager.cc:1397] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:45:44.974670 15415 tablet_replica.cc:439] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } } }
I20260928 06:45:44.974798 15415 tablet_replica.cc:442] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:45:44.974838 15407 raft_consensus.cc:348] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:44.974953 15407 raft_consensus.cc:374] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:44.974997 15407 raft_consensus.cc:729] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2778a9828114f36ae7f9421329e9dfd, State: Initialized, Role: FOLLOWER
I20260928 06:45:44.975138 15407 consensus_queue.cc:260] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:44.975440 15407 ts_tablet_manager.cc:1428] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:45:44.975996 15408 tablet_bootstrap.cc:492] T 4c4a051b514e4f7ea5e108d90db24056 P bd72f0024ada43e7a14d4e1bee99f7e6: No bootstrap required, opened a new log
I20260928 06:45:44.976069 15408 ts_tablet_manager.cc:1397] T 4c4a051b514e4f7ea5e108d90db24056 P bd72f0024ada43e7a14d4e1bee99f7e6: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260928 06:45:44.976540 15408 raft_consensus.cc:348] T 4c4a051b514e4f7ea5e108d90db24056 P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:44.976676 15408 raft_consensus.cc:374] T 4c4a051b514e4f7ea5e108d90db24056 P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:44.976711 15408 raft_consensus.cc:729] T 4c4a051b514e4f7ea5e108d90db24056 P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd72f0024ada43e7a14d4e1bee99f7e6, State: Initialized, Role: FOLLOWER
I20260928 06:45:44.976838 15408 consensus_queue.cc:260] T 4c4a051b514e4f7ea5e108d90db24056 P bd72f0024ada43e7a14d4e1bee99f7e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:44.977067 15414 tablet_replica.cc:439] T 4c4a051b514e4f7ea5e108d90db24056 P bd72f0024ada43e7a14d4e1bee99f7e6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } } }
I20260928 06:45:44.977205 15414 tablet_replica.cc:442] T 4c4a051b514e4f7ea5e108d90db24056 P bd72f0024ada43e7a14d4e1bee99f7e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:45:44.977255 15413 tablet_replica.cc:439] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } } }
I20260928 06:45:44.977378 15413 tablet_replica.cc:442] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:45:44.977118 15408 ts_tablet_manager.cc:1428] T 4c4a051b514e4f7ea5e108d90db24056 P bd72f0024ada43e7a14d4e1bee99f7e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:45:45.288657 15415 raft_consensus.cc:482] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:45:45.288816 15415 raft_consensus.cc:504] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:45.290126 15415 leader_election.cc:288] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd72f0024ada43e7a14d4e1bee99f7e6 (127.7.48.2:40247), f2778a9828114f36ae7f9421329e9dfd (127.7.48.3:36477)
I20260928 06:45:45.295269 15413 raft_consensus.cc:482] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:45:45.295347 15448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:45.290446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:40247 (local address 127.7.48.1:50905)
0928 06:45:45.290811 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:45.290828 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:45.290958 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:45.291442 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:45.291445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:45.291468 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:45.291606 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:45.291613 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.292717 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.292720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:45.294138 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:45.294149 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.294860 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.294863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:45.294975 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:45.294998 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 06:45:45.295097 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 06:45:45.295178 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":8,"thread_start_us":84,"threads_started":1}
I20260928 06:45:45.295333 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:45.290800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:50905 (local address 127.7.48.2:40247)
0928 06:45:45.291181 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:45.291198 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:45.291212 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:45:45.291270 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:45.291273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:45.291323 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:45.291429 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:45.291745 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.292596 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.294279 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.294721 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.294854 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:45.294920 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:45.295115 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 06:45:45.295216 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":174,"threads_started":1}
I20260928 06:45:45.295383 15413 raft_consensus.cc:504] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:45.295816 15413 leader_election.cc:288] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69f4f371eac34d12bca22a7bda991178 (127.7.48.1:36379), bd72f0024ada43e7a14d4e1bee99f7e6 (127.7.48.2:40247)
I20260928 06:45:45.295837 15193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4a051b514e4f7ea5e108d90db24056" candidate_uuid: "69f4f371eac34d12bca22a7bda991178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" is_pre_election: true
I20260928 06:45:45.296023 15193 raft_consensus.cc:2454] T 4c4a051b514e4f7ea5e108d90db24056 P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f4f371eac34d12bca22a7bda991178 in term 0.
I20260928 06:45:45.296185 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:45.290524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:36477 (local address 127.7.48.1:54389)
0928 06:45:45.291154 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:45.291170 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:45.291269 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:45.291940 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:45.291943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:45.291960 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:45:45.292086 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:45.292095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.293144 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.293150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:45.294347 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:45.294355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.295022 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.295026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:45.295145 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:45.295170 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 06:45:45.295725 (+ 555us) client_negotiation.cc:240] Negotiation successful
0928 06:45:45.295807 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":36,"thread_start_us":65,"threads_started":1}
I20260928 06:45:45.296267 15192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4a051b514e4f7ea5e108d90db24056" candidate_uuid: "f2778a9828114f36ae7f9421329e9dfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" is_pre_election: true
I20260928 06:45:45.296387 15192 raft_consensus.cc:2454] T 4c4a051b514e4f7ea5e108d90db24056 P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2778a9828114f36ae7f9421329e9dfd in term 0.
I20260928 06:45:45.296479 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:45.290928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:54389 (local address 127.7.48.3:36477)
0928 06:45:45.291627 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:45.291643 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:45.291661 (+ 18us) server_negotiation.cc:407] Connection header received
0928 06:45:45.291739 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:45.291742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:45.291804 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:45.291949 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:45:45.292241 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.293004 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.294467 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.294890 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.295015 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:45.295080 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:45.295290 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 06:45:45.295375 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607}
I20260928 06:45:45.296414 14984 leader_election.cc:302] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69f4f371eac34d12bca22a7bda991178, bd72f0024ada43e7a14d4e1bee99f7e6; no voters:
I20260928 06:45:45.296705 15415 raft_consensus.cc:2781] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:45:45.296291 15052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4a051b514e4f7ea5e108d90db24056" candidate_uuid: "f2778a9828114f36ae7f9421329e9dfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f4f371eac34d12bca22a7bda991178" is_pre_election: true
I20260928 06:45:45.296766 15415 raft_consensus.cc:482] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:45:45.296784 15264 leader_election.cc:302] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd72f0024ada43e7a14d4e1bee99f7e6, f2778a9828114f36ae7f9421329e9dfd; no voters:
I20260928 06:45:45.296838 15415 raft_consensus.cc:3037] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:45.296958 15413 raft_consensus.cc:2781] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:45:45.296911 15333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4a051b514e4f7ea5e108d90db24056" candidate_uuid: "69f4f371eac34d12bca22a7bda991178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2778a9828114f36ae7f9421329e9dfd" is_pre_election: true
I20260928 06:45:45.297024 15413 raft_consensus.cc:482] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:45:45.297076 15413 raft_consensus.cc:3037] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:45.297976 15413 raft_consensus.cc:504] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:45.297986 15415 raft_consensus.cc:504] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:45.298233 15333 raft_consensus.cc:2379] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f4f371eac34d12bca22a7bda991178 in current term 1: Already voted for candidate f2778a9828114f36ae7f9421329e9dfd in this term.
I20260928 06:45:45.298342 15413 leader_election.cc:288] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd [CANDIDATE]: Term 1 election: Requested vote from peers 69f4f371eac34d12bca22a7bda991178 (127.7.48.1:36379), bd72f0024ada43e7a14d4e1bee99f7e6 (127.7.48.2:40247)
I20260928 06:45:45.298411 15415 leader_election.cc:288] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178 [CANDIDATE]: Term 1 election: Requested vote from peers bd72f0024ada43e7a14d4e1bee99f7e6 (127.7.48.2:40247), f2778a9828114f36ae7f9421329e9dfd (127.7.48.3:36477)
I20260928 06:45:45.298667 15052 raft_consensus.cc:2379] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f2778a9828114f36ae7f9421329e9dfd in current term 1: Already voted for candidate 69f4f371eac34d12bca22a7bda991178 in this term.
I20260928 06:45:45.298771 15053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4a051b514e4f7ea5e108d90db24056" candidate_uuid: "f2778a9828114f36ae7f9421329e9dfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f4f371eac34d12bca22a7bda991178"
I20260928 06:45:45.299044 15192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4a051b514e4f7ea5e108d90db24056" candidate_uuid: "69f4f371eac34d12bca22a7bda991178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6"
I20260928 06:45:45.299170 15192 raft_consensus.cc:3037] T 4c4a051b514e4f7ea5e108d90db24056 P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:45.299399 15333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4a051b514e4f7ea5e108d90db24056" candidate_uuid: "69f4f371eac34d12bca22a7bda991178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2778a9828114f36ae7f9421329e9dfd"
I20260928 06:45:45.299520 15333 raft_consensus.cc:2379] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f4f371eac34d12bca22a7bda991178 in current term 1: Already voted for candidate f2778a9828114f36ae7f9421329e9dfd in this term.
I20260928 06:45:45.300004 15193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4a051b514e4f7ea5e108d90db24056" candidate_uuid: "f2778a9828114f36ae7f9421329e9dfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6"
I20260928 06:45:45.300294 15192 raft_consensus.cc:2454] T 4c4a051b514e4f7ea5e108d90db24056 P bd72f0024ada43e7a14d4e1bee99f7e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f4f371eac34d12bca22a7bda991178 in term 1.
I20260928 06:45:45.300338 15264 leader_election.cc:302] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2778a9828114f36ae7f9421329e9dfd; no voters: 69f4f371eac34d12bca22a7bda991178, bd72f0024ada43e7a14d4e1bee99f7e6
I20260928 06:45:45.300501 15413 raft_consensus.cc:2726] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 06:45:45.300642 14984 leader_election.cc:302] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 69f4f371eac34d12bca22a7bda991178, bd72f0024ada43e7a14d4e1bee99f7e6; no voters: f2778a9828114f36ae7f9421329e9dfd
I20260928 06:45:45.300876 15415 raft_consensus.cc:2781] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:45.301062 15415 raft_consensus.cc:686] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178 [term 1 LEADER]: Becoming Leader. State: Replica: 69f4f371eac34d12bca22a7bda991178, State: Running, Role: LEADER
I20260928 06:45:45.301324 15415 consensus_queue.cc:237] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:45.302444 15451 tablet_replica.cc:439] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178: TxnStatusTablet state changed. Reason: New leader 69f4f371eac34d12bca22a7bda991178. Latest consensus state: current_term: 1 leader_uuid: "69f4f371eac34d12bca22a7bda991178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } } }
I20260928 06:45:45.302587 15451 tablet_replica.cc:442] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:45:45.302860 15453 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 06:45:45.303010 15453 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 06:45:45.303678 14895 catalog_manager.cc:5495] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69f4f371eac34d12bca22a7bda991178 (127.7.48.1). New cstate: current_term: 1 leader_uuid: "69f4f371eac34d12bca22a7bda991178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } health_report { overall_health: UNKNOWN } } }
I20260928 06:45:45.345628 15406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:45.341509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54562 (local address 127.7.48.1:36379)
0928 06:45:45.341651 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:45.341656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:45.341671 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:45:45.341797 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:45.341801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:45.341857 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:45.341972 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:45:45.342221 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.342833 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.344174 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.344396 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.344477 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:45.344816 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:45:45.345237 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:45:45.345323 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:45.345461 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 06:45:45.345514 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 06:45:45.345672 15444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:45.341361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:36379 (local address 127.0.0.1:54562)
0928 06:45:45.341487 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:45.341504 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:45.341642 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:45.341985 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:45.341988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:45.341998 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:45:45.342111 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:45.342116 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.343007 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.343011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:45.344049 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:45.344062 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.344510 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.344514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:45.344614 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:45.344668 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:45:45.345335 (+ 667us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:45:45.345350 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 06:45:45.345439 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:45:45.345531 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 06:45:45.347715 15414 consensus_queue.cc:1035] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:45.348735 15192 raft_consensus.cc:1261] T 4c4a051b514e4f7ea5e108d90db24056 P bd72f0024ada43e7a14d4e1bee99f7e6 [term 1 FOLLOWER]: Refusing update from remote peer 69f4f371eac34d12bca22a7bda991178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:45.349205 15332 raft_consensus.cc:1261] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd [term 1 FOLLOWER]: Refusing update from remote peer 69f4f371eac34d12bca22a7bda991178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:45.349622 15451 consensus_queue.cc:1035] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:45.349822 15415 consensus_queue.cc:1035] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:45.351306 15413 tablet_replica.cc:439] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd: TxnStatusTablet state changed. Reason: New leader 69f4f371eac34d12bca22a7bda991178. Latest consensus state: current_term: 1 leader_uuid: "69f4f371eac34d12bca22a7bda991178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } } }
I20260928 06:45:45.351432 15413 tablet_replica.cc:442] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:45:45.352245 15437 tablet_replica.cc:439] T 4c4a051b514e4f7ea5e108d90db24056 P bd72f0024ada43e7a14d4e1bee99f7e6: TxnStatusTablet state changed. Reason: New leader 69f4f371eac34d12bca22a7bda991178. Latest consensus state: current_term: 1 leader_uuid: "69f4f371eac34d12bca22a7bda991178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } } }
I20260928 06:45:45.352361 15437 tablet_replica.cc:442] T 4c4a051b514e4f7ea5e108d90db24056 P bd72f0024ada43e7a14d4e1bee99f7e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:45:45.352941 15462 consensus_queue.cc:1035] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:45.353722 15415 tablet_replica.cc:439] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "69f4f371eac34d12bca22a7bda991178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } } }
I20260928 06:45:45.353826 15415 tablet_replica.cc:442] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:45:45.353971 15456 mvcc.cc:205] Tried to move back new op lower bound from 7334207264143560704 to 7334207263956545536. Current Snapshot: MvccSnapshot[applied={T|T < 7334207264143560704}]
I20260928 06:45:45.355234 15451 tablet_replica.cc:439] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "69f4f371eac34d12bca22a7bda991178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } } }
I20260928 06:45:45.355330 15451 tablet_replica.cc:442] T 4c4a051b514e4f7ea5e108d90db24056 P 69f4f371eac34d12bca22a7bda991178: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:45:45.356038 15413 tablet_replica.cc:439] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "69f4f371eac34d12bca22a7bda991178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } } }
I20260928 06:45:45.356086 15414 tablet_replica.cc:439] T 4c4a051b514e4f7ea5e108d90db24056 P bd72f0024ada43e7a14d4e1bee99f7e6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "69f4f371eac34d12bca22a7bda991178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } } }
I20260928 06:45:45.356146 15413 tablet_replica.cc:442] T 4c4a051b514e4f7ea5e108d90db24056 P f2778a9828114f36ae7f9421329e9dfd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:45:45.356177 15414 tablet_replica.cc:442] T 4c4a051b514e4f7ea5e108d90db24056 P bd72f0024ada43e7a14d4e1bee99f7e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:45:45.366413 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:45.362538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58434 (local address 127.7.48.2:40247)
0928 06:45:45.362675 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:45.362680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:45.362695 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:45:45.362787 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:45.362790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:45.362849 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:45.362986 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:45:45.363192 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.363998 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.364830 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.365170 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.365271 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:45.365501 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:45:45.366059 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:45:45.366174 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:45.366215 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 06:45:45.366282 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 06:45:45.377347 15406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:45.373917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:38213 (local address 127.7.48.1:36379)
0928 06:45:45.374048 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:45.374051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:45.374178 (+ 127us) server_negotiation.cc:407] Connection header received
0928 06:45:45.374312 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:45.374315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:45.374358 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:45.374414 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:45:45.374691 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.375166 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.376381 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.376578 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.376653 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:45.376969 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:45:45.377031 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:45:45.377081 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:45.377201 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 06:45:45.377243 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:45:45.377429 15475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:45.373755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:36379 (local address 127.7.48.2:38213)
0928 06:45:45.374091 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:45.374106 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:45.374208 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:45.374454 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:45.374457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:45.374465 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:45:45.374591 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:45.374597 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.375273 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.375276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:45.376270 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:45.376281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.376702 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.376706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:45.376799 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:45.376848 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:45:45.377109 (+ 261us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:45:45.377116 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 06:45:45.377185 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:45:45.377263 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":125,"threads_started":1}
I20260928 06:45:45.403352 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:45.399474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:52897 (local address 127.7.48.2:40247)
0928 06:45:45.399686 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:45.399691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:45.399804 (+ 113us) server_negotiation.cc:407] Connection header received
0928 06:45:45.399914 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:45.399916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:45.399966 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:45.400047 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:45:45.400341 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.401118 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.402210 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.402566 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.402670 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:45.402926 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:45:45.403021 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:45:45.403106 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:45.403175 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 06:45:45.403227 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260928 06:45:45.403352 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:45.399407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:40247 (local address 127.7.48.1:52897)
0928 06:45:45.399719 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:45.399732 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:45.399817 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:45.400054 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:45.400056 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:45.400062 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:45:45.400181 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:45.400187 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.401211 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.401214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:45.402087 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:45.402096 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.402682 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.402684 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:45.402758 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:45.402800 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:45:45.403108 (+ 308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:45:45.403112 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:45:45.403162 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:45:45.403223 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":107,"threads_started":1}
I20260928 06:45:45.430615 14896 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:45.430811 14896 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:45.433316 14896 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc: Sending DeleteTablet for 3 replicas of tablet 32afe850044947f78fd909ac6177389e
I20260928 06:45:45.434018 15313 tablet_service.cc:1514] Processing DeleteTablet for tablet 32afe850044947f78fd909ac6177389e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:45 UTC) from {username='oryx', principal='oryx/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:59870
I20260928 06:45:45.434648 15033 tablet_service.cc:1514] Processing DeleteTablet for tablet 32afe850044947f78fd909ac6177389e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:45 UTC) from {username='oryx', principal='oryx/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:60448
I20260928 06:45:45.435135 15480 tablet_replica.cc:330] T 32afe850044947f78fd909ac6177389e P f2778a9828114f36ae7f9421329e9dfd: stopping tablet replica
I20260928 06:45:45.435367 15481 tablet_replica.cc:330] T 32afe850044947f78fd909ac6177389e P 69f4f371eac34d12bca22a7bda991178: stopping tablet replica
I20260928 06:45:45.435555 15481 raft_consensus.cc:2229] T 32afe850044947f78fd909ac6177389e P 69f4f371eac34d12bca22a7bda991178 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:45.435750 15480 raft_consensus.cc:2229] T 32afe850044947f78fd909ac6177389e P f2778a9828114f36ae7f9421329e9dfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:45.435971 15481 raft_consensus.cc:2258] T 32afe850044947f78fd909ac6177389e P 69f4f371eac34d12bca22a7bda991178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:45.436174 15480 raft_consensus.cc:2258] T 32afe850044947f78fd909ac6177389e P f2778a9828114f36ae7f9421329e9dfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:45.437052 15480 ts_tablet_manager.cc:1905] T 32afe850044947f78fd909ac6177389e P f2778a9828114f36ae7f9421329e9dfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:45.437762 15481 ts_tablet_manager.cc:1905] T 32afe850044947f78fd909ac6177389e P 69f4f371eac34d12bca22a7bda991178: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:45.441013 15481 ts_tablet_manager.cc:1918] T 32afe850044947f78fd909ac6177389e P 69f4f371eac34d12bca22a7bda991178: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:45:45.441356 15173 tablet_service.cc:1514] Processing DeleteTablet for tablet 32afe850044947f78fd909ac6177389e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:45 UTC) from {username='oryx', principal='oryx/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:39484
I20260928 06:45:45.444691 15480 ts_tablet_manager.cc:1918] T 32afe850044947f78fd909ac6177389e P f2778a9828114f36ae7f9421329e9dfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:45:45.444855 15481 log.cc:1196] T 32afe850044947f78fd909ac6177389e P 69f4f371eac34d12bca22a7bda991178: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-0/wal/wals/32afe850044947f78fd909ac6177389e
I20260928 06:45:45.445258 15481 ts_tablet_manager.cc:1939] T 32afe850044947f78fd909ac6177389e P 69f4f371eac34d12bca22a7bda991178: Deleting consensus metadata
I20260928 06:45:45.445403 15480 log.cc:1196] T 32afe850044947f78fd909ac6177389e P f2778a9828114f36ae7f9421329e9dfd: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/32afe850044947f78fd909ac6177389e
I20260928 06:45:45.445768 15480 ts_tablet_manager.cc:1939] T 32afe850044947f78fd909ac6177389e P f2778a9828114f36ae7f9421329e9dfd: Deleting consensus metadata
I20260928 06:45:45.446707 14882 catalog_manager.cc:4841] TS 69f4f371eac34d12bca22a7bda991178 (127.7.48.1:36379): tablet 32afe850044947f78fd909ac6177389e (table test-table [id=f41fb61db2a74b8ca9ae898e174c929b]) successfully deleted
I20260928 06:45:45.448123 14882 catalog_manager.cc:4841] TS f2778a9828114f36ae7f9421329e9dfd (127.7.48.3:36477): tablet 32afe850044947f78fd909ac6177389e (table test-table [id=f41fb61db2a74b8ca9ae898e174c929b]) successfully deleted
I20260928 06:45:45.452153 15483 tablet_replica.cc:330] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6: stopping tablet replica
I20260928 06:45:45.453887 15483 raft_consensus.cc:2229] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:45:45.454334 15483 raft_consensus.cc:2258] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:45.455190 15483 ts_tablet_manager.cc:1905] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:45.458455 15483 ts_tablet_manager.cc:1918] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:45:45.458599 15483 log.cc:1196] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/32afe850044947f78fd909ac6177389e
I20260928 06:45:45.458956 15483 ts_tablet_manager.cc:1939] T 32afe850044947f78fd909ac6177389e P bd72f0024ada43e7a14d4e1bee99f7e6: Deleting consensus metadata
I20260928 06:45:45.459765 14880 catalog_manager.cc:4841] TS bd72f0024ada43e7a14d4e1bee99f7e6 (127.7.48.2:40247): tablet 32afe850044947f78fd909ac6177389e (table test-table [id=f41fb61db2a74b8ca9ae898e174c929b]) successfully deleted
Sep 28 06:45:45 dist-test-slave-twl6 krb5kdc[14851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:45:45 dist-test-slave-twl6 krb5kdc[14851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:45 dist-test-slave-twl6 krb5kdc[14851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:45:45.492669 15490 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:45.480872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59464 (local address 127.7.48.62:46109)
0928 06:45:45.481118 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:45.481123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:45.481136 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:45:45.481185 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:45.481188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:45.481236 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:45.481321 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:45.482236 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.483094 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.484459 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.484801 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.484916 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:45.492508 (+ 7592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59464: BlockingRecv error: recv got EOF from 127.0.0.1:59464 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":60,"threads_started":1}
Sep 28 06:45:45 dist-test-slave-twl6 krb5kdc[14851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM
I20260928 06:45:45.502964 15490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:45.494262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59474 (local address 127.7.48.62:46109)
0928 06:45:45.494389 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:45.494398 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:45.494493 (+ 95us) server_negotiation.cc:407] Connection header received
0928 06:45:45.494620 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:45.494622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:45.494665 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:45.494732 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:45.495451 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.496038 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.497164 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.497369 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.497445 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:45.499813 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:45.499816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:45.499818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:45.499844 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:45.501344 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:45.501753 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:45.501757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:45.501759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:45.501839 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:45.502114 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:45.502117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:45.502119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:45.502293 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:45.502433 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:45.502672 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 06:45:45.502796 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 06:45:45.505661 14896 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:45:45.506124 14896 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:45:45.511030 15173 tablet_service.cc:1467] Processing CreateTablet for tablet 7bc3ba54a2fc4a59a319db62680893bd (DEFAULT_TABLE table=test-table [id=6a9e7fb2f1e740828ed4fb66c1efe18b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:45.511343 15173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bc3ba54a2fc4a59a319db62680893bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:45.511303 15313 tablet_service.cc:1467] Processing CreateTablet for tablet 7bc3ba54a2fc4a59a319db62680893bd (DEFAULT_TABLE table=test-table [id=6a9e7fb2f1e740828ed4fb66c1efe18b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:45.511584 15313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bc3ba54a2fc4a59a319db62680893bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:45.512173 15033 tablet_service.cc:1467] Processing CreateTablet for tablet 7bc3ba54a2fc4a59a319db62680893bd (DEFAULT_TABLE table=test-table [id=6a9e7fb2f1e740828ed4fb66c1efe18b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:45.512423 15033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bc3ba54a2fc4a59a319db62680893bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:45.514003 15496 tablet_bootstrap.cc:492] T 7bc3ba54a2fc4a59a319db62680893bd P bd72f0024ada43e7a14d4e1bee99f7e6: Bootstrap starting.
I20260928 06:45:45.514631 15497 tablet_bootstrap.cc:492] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd: Bootstrap starting.
I20260928 06:45:45.515012 15496 tablet_bootstrap.cc:654] T 7bc3ba54a2fc4a59a319db62680893bd P bd72f0024ada43e7a14d4e1bee99f7e6: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:45.516090 15497 tablet_bootstrap.cc:654] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:45.516204 15496 tablet_bootstrap.cc:492] T 7bc3ba54a2fc4a59a319db62680893bd P bd72f0024ada43e7a14d4e1bee99f7e6: No bootstrap required, opened a new log
I20260928 06:45:45.516296 15496 ts_tablet_manager.cc:1397] T 7bc3ba54a2fc4a59a319db62680893bd P bd72f0024ada43e7a14d4e1bee99f7e6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:45.516727 15496 raft_consensus.cc:348] T 7bc3ba54a2fc4a59a319db62680893bd P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:45.516839 15496 raft_consensus.cc:374] T 7bc3ba54a2fc4a59a319db62680893bd P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:45.516873 15496 raft_consensus.cc:729] T 7bc3ba54a2fc4a59a319db62680893bd P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd72f0024ada43e7a14d4e1bee99f7e6, State: Initialized, Role: FOLLOWER
I20260928 06:45:45.516990 15496 consensus_queue.cc:260] T 7bc3ba54a2fc4a59a319db62680893bd P bd72f0024ada43e7a14d4e1bee99f7e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:45.517239 15496 ts_tablet_manager.cc:1428] T 7bc3ba54a2fc4a59a319db62680893bd P bd72f0024ada43e7a14d4e1bee99f7e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:45:45.517274 15497 tablet_bootstrap.cc:492] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd: No bootstrap required, opened a new log
I20260928 06:45:45.517336 15497 ts_tablet_manager.cc:1397] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:45:45.517685 15497 raft_consensus.cc:348] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:45.517805 15497 raft_consensus.cc:374] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:45.517889 15497 raft_consensus.cc:729] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2778a9828114f36ae7f9421329e9dfd, State: Initialized, Role: FOLLOWER
I20260928 06:45:45.518055 15497 consensus_queue.cc:260] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:45.518800 15497 ts_tablet_manager.cc:1428] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:45:45.520082 15498 tablet_bootstrap.cc:492] T 7bc3ba54a2fc4a59a319db62680893bd P 69f4f371eac34d12bca22a7bda991178: Bootstrap starting.
I20260928 06:45:45.521605 15498 tablet_bootstrap.cc:654] T 7bc3ba54a2fc4a59a319db62680893bd P 69f4f371eac34d12bca22a7bda991178: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:45.525431 15498 tablet_bootstrap.cc:492] T 7bc3ba54a2fc4a59a319db62680893bd P 69f4f371eac34d12bca22a7bda991178: No bootstrap required, opened a new log
I20260928 06:45:45.525563 15498 ts_tablet_manager.cc:1397] T 7bc3ba54a2fc4a59a319db62680893bd P 69f4f371eac34d12bca22a7bda991178: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 06:45:45.526021 15498 raft_consensus.cc:348] T 7bc3ba54a2fc4a59a319db62680893bd P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:45.526141 15498 raft_consensus.cc:374] T 7bc3ba54a2fc4a59a319db62680893bd P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:45.526185 15498 raft_consensus.cc:729] T 7bc3ba54a2fc4a59a319db62680893bd P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f4f371eac34d12bca22a7bda991178, State: Initialized, Role: FOLLOWER
I20260928 06:45:45.526304 15498 consensus_queue.cc:260] T 7bc3ba54a2fc4a59a319db62680893bd P 69f4f371eac34d12bca22a7bda991178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:45.526584 15498 ts_tablet_manager.cc:1428] T 7bc3ba54a2fc4a59a319db62680893bd P 69f4f371eac34d12bca22a7bda991178: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:45:45.740928 15413 raft_consensus.cc:482] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:45:45.741098 15413 raft_consensus.cc:504] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:45.741570 15413 leader_election.cc:288] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69f4f371eac34d12bca22a7bda991178 (127.7.48.1:36379), bd72f0024ada43e7a14d4e1bee99f7e6 (127.7.48.2:40247)
I20260928 06:45:45.742077 15052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bc3ba54a2fc4a59a319db62680893bd" candidate_uuid: "f2778a9828114f36ae7f9421329e9dfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f4f371eac34d12bca22a7bda991178" is_pre_election: true
I20260928 06:45:45.742115 15192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bc3ba54a2fc4a59a319db62680893bd" candidate_uuid: "f2778a9828114f36ae7f9421329e9dfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" is_pre_election: true
I20260928 06:45:45.742220 15052 raft_consensus.cc:2454] T 7bc3ba54a2fc4a59a319db62680893bd P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2778a9828114f36ae7f9421329e9dfd in term 0.
I20260928 06:45:45.742295 15192 raft_consensus.cc:2454] T 7bc3ba54a2fc4a59a319db62680893bd P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2778a9828114f36ae7f9421329e9dfd in term 0.
I20260928 06:45:45.742619 15266 leader_election.cc:302] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 69f4f371eac34d12bca22a7bda991178, bd72f0024ada43e7a14d4e1bee99f7e6, f2778a9828114f36ae7f9421329e9dfd; no voters:
I20260928 06:45:45.742822 15413 raft_consensus.cc:2781] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:45:45.742924 15413 raft_consensus.cc:482] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:45:45.742959 15413 raft_consensus.cc:3037] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:45.744113 15413 raft_consensus.cc:504] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:45.744468 15413 leader_election.cc:288] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd [CANDIDATE]: Term 1 election: Requested vote from peers 69f4f371eac34d12bca22a7bda991178 (127.7.48.1:36379), bd72f0024ada43e7a14d4e1bee99f7e6 (127.7.48.2:40247)
I20260928 06:45:45.744822 15052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bc3ba54a2fc4a59a319db62680893bd" candidate_uuid: "f2778a9828114f36ae7f9421329e9dfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f4f371eac34d12bca22a7bda991178"
I20260928 06:45:45.744848 15192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bc3ba54a2fc4a59a319db62680893bd" candidate_uuid: "f2778a9828114f36ae7f9421329e9dfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6"
I20260928 06:45:45.744925 15052 raft_consensus.cc:3037] T 7bc3ba54a2fc4a59a319db62680893bd P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:45.744974 15192 raft_consensus.cc:3037] T 7bc3ba54a2fc4a59a319db62680893bd P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:45.745901 15052 raft_consensus.cc:2454] T 7bc3ba54a2fc4a59a319db62680893bd P 69f4f371eac34d12bca22a7bda991178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2778a9828114f36ae7f9421329e9dfd in term 1.
I20260928 06:45:45.746028 15192 raft_consensus.cc:2454] T 7bc3ba54a2fc4a59a319db62680893bd P bd72f0024ada43e7a14d4e1bee99f7e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2778a9828114f36ae7f9421329e9dfd in term 1.
I20260928 06:45:45.746268 15266 leader_election.cc:302] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69f4f371eac34d12bca22a7bda991178, f2778a9828114f36ae7f9421329e9dfd; no voters:
I20260928 06:45:45.746539 15413 raft_consensus.cc:2781] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:45.746724 15413 raft_consensus.cc:686] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd [term 1 LEADER]: Becoming Leader. State: Replica: f2778a9828114f36ae7f9421329e9dfd, State: Running, Role: LEADER
I20260928 06:45:45.746878 15413 consensus_queue.cc:237] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:45.748764 14895 catalog_manager.cc:5495] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd reported cstate change: term changed from 0 to 1, leader changed from <none> to f2778a9828114f36ae7f9421329e9dfd (127.7.48.3). New cstate: current_term: 1 leader_uuid: "f2778a9828114f36ae7f9421329e9dfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } health_report { overall_health: HEALTHY } } }
I20260928 06:45:45.811065 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:45.807481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35524 (local address 127.7.48.3:36477)
0928 06:45:45.807818 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:45.807823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:45.807838 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:45:45.807927 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:45.807930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:45.808009 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:45.808177 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:45:45.808538 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.809070 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.809955 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:45.810206 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:45.810314 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:45.810581 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:45:45.810667 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:45:45.810750 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:45.810841 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 06:45:45.810899 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":103,"threads_started":1}
I20260928 06:45:45.813097 15052 raft_consensus.cc:1261] T 7bc3ba54a2fc4a59a319db62680893bd P 69f4f371eac34d12bca22a7bda991178 [term 1 FOLLOWER]: Refusing update from remote peer f2778a9828114f36ae7f9421329e9dfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:45.813107 15192 raft_consensus.cc:1261] T 7bc3ba54a2fc4a59a319db62680893bd P bd72f0024ada43e7a14d4e1bee99f7e6 [term 1 FOLLOWER]: Refusing update from remote peer f2778a9828114f36ae7f9421329e9dfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:45.813613 15413 consensus_queue.cc:1035] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:45.813803 15502 consensus_queue.cc:1035] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:45.821241 14895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:45.821369 14895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:45.823407 14895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc: Sending DeleteTablet for 3 replicas of tablet 7bc3ba54a2fc4a59a319db62680893bd
I20260928 06:45:45.823948 15033 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bc3ba54a2fc4a59a319db62680893bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:45 UTC) from {username='oryx', principal='oryx/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:60448
I20260928 06:45:45.823992 15173 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bc3ba54a2fc4a59a319db62680893bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:45 UTC) from {username='oryx', principal='oryx/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:39484
I20260928 06:45:45.824025 15313 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bc3ba54a2fc4a59a319db62680893bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:45 UTC) from {username='oryx', principal='oryx/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:59870
I20260928 06:45:45.824127 15481 tablet_replica.cc:330] T 7bc3ba54a2fc4a59a319db62680893bd P 69f4f371eac34d12bca22a7bda991178: stopping tablet replica
I20260928 06:45:45.824164 15483 tablet_replica.cc:330] T 7bc3ba54a2fc4a59a319db62680893bd P bd72f0024ada43e7a14d4e1bee99f7e6: stopping tablet replica
I20260928 06:45:45.824194 15480 tablet_replica.cc:330] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd: stopping tablet replica
I20260928 06:45:45.824239 15481 raft_consensus.cc:2229] T 7bc3ba54a2fc4a59a319db62680893bd P 69f4f371eac34d12bca22a7bda991178 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:45.824251 15483 raft_consensus.cc:2229] T 7bc3ba54a2fc4a59a319db62680893bd P bd72f0024ada43e7a14d4e1bee99f7e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:45.824301 15480 raft_consensus.cc:2229] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:45:45.824357 15481 raft_consensus.cc:2258] T 7bc3ba54a2fc4a59a319db62680893bd P 69f4f371eac34d12bca22a7bda991178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:45.824357 15483 raft_consensus.cc:2258] T 7bc3ba54a2fc4a59a319db62680893bd P bd72f0024ada43e7a14d4e1bee99f7e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:45.824489 15480 raft_consensus.cc:2258] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:45.824862 14895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 06:45:45.825063 15483 ts_tablet_manager.cc:1905] T 7bc3ba54a2fc4a59a319db62680893bd P bd72f0024ada43e7a14d4e1bee99f7e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:45.825083 15480 ts_tablet_manager.cc:1905] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:45.825170 15481 ts_tablet_manager.cc:1905] T 7bc3ba54a2fc4a59a319db62680893bd P 69f4f371eac34d12bca22a7bda991178: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:45:45.825497 14895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:45:45.827419 15481 ts_tablet_manager.cc:1918] T 7bc3ba54a2fc4a59a319db62680893bd P 69f4f371eac34d12bca22a7bda991178: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:45:45.827419 15483 ts_tablet_manager.cc:1918] T 7bc3ba54a2fc4a59a319db62680893bd P bd72f0024ada43e7a14d4e1bee99f7e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:45:45.827504 15481 log.cc:1196] T 7bc3ba54a2fc4a59a319db62680893bd P 69f4f371eac34d12bca22a7bda991178: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-0/wal/wals/7bc3ba54a2fc4a59a319db62680893bd
I20260928 06:45:45.827504 15483 log.cc:1196] T 7bc3ba54a2fc4a59a319db62680893bd P bd72f0024ada43e7a14d4e1bee99f7e6: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/7bc3ba54a2fc4a59a319db62680893bd
I20260928 06:45:45.827630 15480 ts_tablet_manager.cc:1918] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:45:45.827700 15480 log.cc:1196] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/7bc3ba54a2fc4a59a319db62680893bd
I20260928 06:45:45.827769 15483 ts_tablet_manager.cc:1939] T 7bc3ba54a2fc4a59a319db62680893bd P bd72f0024ada43e7a14d4e1bee99f7e6: Deleting consensus metadata
I20260928 06:45:45.827769 15481 ts_tablet_manager.cc:1939] T 7bc3ba54a2fc4a59a319db62680893bd P 69f4f371eac34d12bca22a7bda991178: Deleting consensus metadata
I20260928 06:45:45.828078 15480 ts_tablet_manager.cc:1939] T 7bc3ba54a2fc4a59a319db62680893bd P f2778a9828114f36ae7f9421329e9dfd: Deleting consensus metadata
I20260928 06:45:45.828497 14882 catalog_manager.cc:4841] TS 69f4f371eac34d12bca22a7bda991178 (127.7.48.1:36379): tablet 7bc3ba54a2fc4a59a319db62680893bd (table test-table [id=6a9e7fb2f1e740828ed4fb66c1efe18b]) successfully deleted
I20260928 06:45:45.828673 14882 catalog_manager.cc:4841] TS f2778a9828114f36ae7f9421329e9dfd (127.7.48.3:36477): tablet 7bc3ba54a2fc4a59a319db62680893bd (table test-table [id=6a9e7fb2f1e740828ed4fb66c1efe18b]) successfully deleted
I20260928 06:45:45.829216 14880 catalog_manager.cc:4841] TS bd72f0024ada43e7a14d4e1bee99f7e6 (127.7.48.2:40247): tablet 7bc3ba54a2fc4a59a319db62680893bd (table test-table [id=6a9e7fb2f1e740828ed4fb66c1efe18b]) successfully deleted
I20260928 06:45:45.830299 15033 tablet_service.cc:1467] Processing CreateTablet for tablet 67097c2134cb485883d82af055d8e596 (DEFAULT_TABLE table=test-table [id=0825613668b3455c8996d413f6c05bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:45.830299 15173 tablet_service.cc:1467] Processing CreateTablet for tablet 67097c2134cb485883d82af055d8e596 (DEFAULT_TABLE table=test-table [id=0825613668b3455c8996d413f6c05bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:45.830407 15313 tablet_service.cc:1467] Processing CreateTablet for tablet 67097c2134cb485883d82af055d8e596 (DEFAULT_TABLE table=test-table [id=0825613668b3455c8996d413f6c05bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:45:45.830525 15173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67097c2134cb485883d82af055d8e596. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:45.830526 15033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67097c2134cb485883d82af055d8e596. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:45.830597 15313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67097c2134cb485883d82af055d8e596. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:45.832674 15498 tablet_bootstrap.cc:492] T 67097c2134cb485883d82af055d8e596 P 69f4f371eac34d12bca22a7bda991178: Bootstrap starting.
I20260928 06:45:45.832813 15497 tablet_bootstrap.cc:492] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd: Bootstrap starting.
I20260928 06:45:45.833587 15498 tablet_bootstrap.cc:654] T 67097c2134cb485883d82af055d8e596 P 69f4f371eac34d12bca22a7bda991178: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:45.833587 15497 tablet_bootstrap.cc:654] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:45.834146 15496 tablet_bootstrap.cc:492] T 67097c2134cb485883d82af055d8e596 P bd72f0024ada43e7a14d4e1bee99f7e6: Bootstrap starting.
I20260928 06:45:45.834724 15497 tablet_bootstrap.cc:492] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd: No bootstrap required, opened a new log
I20260928 06:45:45.834811 15497 ts_tablet_manager.cc:1397] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:45:45.834982 15496 tablet_bootstrap.cc:654] T 67097c2134cb485883d82af055d8e596 P bd72f0024ada43e7a14d4e1bee99f7e6: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:45.835234 15497 raft_consensus.cc:348] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:45.835350 15497 raft_consensus.cc:374] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:45.835397 15497 raft_consensus.cc:729] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2778a9828114f36ae7f9421329e9dfd, State: Initialized, Role: FOLLOWER
I20260928 06:45:45.835554 15497 consensus_queue.cc:260] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:45.835858 15497 ts_tablet_manager.cc:1428] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:45:45.836073 15498 tablet_bootstrap.cc:492] T 67097c2134cb485883d82af055d8e596 P 69f4f371eac34d12bca22a7bda991178: No bootstrap required, opened a new log
I20260928 06:45:45.836098 15496 tablet_bootstrap.cc:492] T 67097c2134cb485883d82af055d8e596 P bd72f0024ada43e7a14d4e1bee99f7e6: No bootstrap required, opened a new log
I20260928 06:45:45.836158 15498 ts_tablet_manager.cc:1397] T 67097c2134cb485883d82af055d8e596 P 69f4f371eac34d12bca22a7bda991178: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 06:45:45.836199 15496 ts_tablet_manager.cc:1397] T 67097c2134cb485883d82af055d8e596 P bd72f0024ada43e7a14d4e1bee99f7e6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:45.836521 15498 raft_consensus.cc:348] T 67097c2134cb485883d82af055d8e596 P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:45.836637 15498 raft_consensus.cc:374] T 67097c2134cb485883d82af055d8e596 P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:45.836602 15496 raft_consensus.cc:348] T 67097c2134cb485883d82af055d8e596 P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:45.836680 15498 raft_consensus.cc:729] T 67097c2134cb485883d82af055d8e596 P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f4f371eac34d12bca22a7bda991178, State: Initialized, Role: FOLLOWER
I20260928 06:45:45.836714 15496 raft_consensus.cc:374] T 67097c2134cb485883d82af055d8e596 P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:45.836747 15496 raft_consensus.cc:729] T 67097c2134cb485883d82af055d8e596 P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd72f0024ada43e7a14d4e1bee99f7e6, State: Initialized, Role: FOLLOWER
I20260928 06:45:45.836786 15498 consensus_queue.cc:260] T 67097c2134cb485883d82af055d8e596 P 69f4f371eac34d12bca22a7bda991178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:45.836881 15496 consensus_queue.cc:260] T 67097c2134cb485883d82af055d8e596 P bd72f0024ada43e7a14d4e1bee99f7e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:45.837064 15498 ts_tablet_manager.cc:1428] T 67097c2134cb485883d82af055d8e596 P 69f4f371eac34d12bca22a7bda991178: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:45:45.837131 15496 ts_tablet_manager.cc:1428] T 67097c2134cb485883d82af055d8e596 P bd72f0024ada43e7a14d4e1bee99f7e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:45:46.144707 15502 raft_consensus.cc:482] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:45:46.144891 15502 raft_consensus.cc:504] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:46.145380 15502 leader_election.cc:288] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69f4f371eac34d12bca22a7bda991178 (127.7.48.1:36379), bd72f0024ada43e7a14d4e1bee99f7e6 (127.7.48.2:40247)
I20260928 06:45:46.145891 15052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67097c2134cb485883d82af055d8e596" candidate_uuid: "f2778a9828114f36ae7f9421329e9dfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f4f371eac34d12bca22a7bda991178" is_pre_election: true
I20260928 06:45:46.145889 15192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67097c2134cb485883d82af055d8e596" candidate_uuid: "f2778a9828114f36ae7f9421329e9dfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" is_pre_election: true
I20260928 06:45:46.146036 15192 raft_consensus.cc:2454] T 67097c2134cb485883d82af055d8e596 P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2778a9828114f36ae7f9421329e9dfd in term 0.
I20260928 06:45:46.146078 15052 raft_consensus.cc:2454] T 67097c2134cb485883d82af055d8e596 P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2778a9828114f36ae7f9421329e9dfd in term 0.
I20260928 06:45:46.146379 15264 leader_election.cc:302] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd72f0024ada43e7a14d4e1bee99f7e6, f2778a9828114f36ae7f9421329e9dfd; no voters:
I20260928 06:45:46.146556 15502 raft_consensus.cc:2781] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:45:46.146642 15502 raft_consensus.cc:482] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:45:46.146699 15502 raft_consensus.cc:3037] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:46.147727 15502 raft_consensus.cc:504] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:46.148115 15502 leader_election.cc:288] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd [CANDIDATE]: Term 1 election: Requested vote from peers 69f4f371eac34d12bca22a7bda991178 (127.7.48.1:36379), bd72f0024ada43e7a14d4e1bee99f7e6 (127.7.48.2:40247)
I20260928 06:45:46.148500 15052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67097c2134cb485883d82af055d8e596" candidate_uuid: "f2778a9828114f36ae7f9421329e9dfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f4f371eac34d12bca22a7bda991178"
I20260928 06:45:46.148505 15192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67097c2134cb485883d82af055d8e596" candidate_uuid: "f2778a9828114f36ae7f9421329e9dfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6"
I20260928 06:45:46.148665 15192 raft_consensus.cc:3037] T 67097c2134cb485883d82af055d8e596 P bd72f0024ada43e7a14d4e1bee99f7e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:46.148676 15052 raft_consensus.cc:3037] T 67097c2134cb485883d82af055d8e596 P 69f4f371eac34d12bca22a7bda991178 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:46.149617 15052 raft_consensus.cc:2454] T 67097c2134cb485883d82af055d8e596 P 69f4f371eac34d12bca22a7bda991178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2778a9828114f36ae7f9421329e9dfd in term 1.
I20260928 06:45:46.149639 15192 raft_consensus.cc:2454] T 67097c2134cb485883d82af055d8e596 P bd72f0024ada43e7a14d4e1bee99f7e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2778a9828114f36ae7f9421329e9dfd in term 1.
I20260928 06:45:46.149978 15266 leader_election.cc:302] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 69f4f371eac34d12bca22a7bda991178, bd72f0024ada43e7a14d4e1bee99f7e6, f2778a9828114f36ae7f9421329e9dfd; no voters:
I20260928 06:45:46.150198 15502 raft_consensus.cc:2781] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:46.150312 15502 raft_consensus.cc:686] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd [term 1 LEADER]: Becoming Leader. State: Replica: f2778a9828114f36ae7f9421329e9dfd, State: Running, Role: LEADER
I20260928 06:45:46.150453 15502 consensus_queue.cc:237] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } }
I20260928 06:45:46.152259 14894 catalog_manager.cc:5495] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd reported cstate change: term changed from 0 to 1, leader changed from <none> to f2778a9828114f36ae7f9421329e9dfd (127.7.48.3). New cstate: current_term: 1 leader_uuid: "f2778a9828114f36ae7f9421329e9dfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2778a9828114f36ae7f9421329e9dfd" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36477 } health_report { overall_health: HEALTHY } } }
I20260928 06:45:46.214825 15517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:46.211499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:36379 (local address 127.7.48.3:53869)
0928 06:45:46.211814 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:46.211835 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:46.211980 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:46.212241 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:46.212243 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:46.212249 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:45:46.212357 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:46.212361 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:46.213050 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:46.213052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:46.213704 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:46.213712 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:46.214216 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:46.214218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:46.214286 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:46.214331 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:45:46.214576 (+ 245us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:45:46.214581 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:45:46.214628 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 06:45:46.214700 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":123,"threads_started":1}
I20260928 06:45:46.214886 15518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:46.211620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:53869 (local address 127.7.48.1:36379)
0928 06:45:46.211860 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:46.211863 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:46.211942 (+ 79us) server_negotiation.cc:407] Connection header received
0928 06:45:46.212085 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:46.212087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:46.212145 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:46.212218 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:45:46.212451 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:46.212953 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:46.213827 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:46.214113 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:46.214196 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:46.214430 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:45:46.214504 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:45:46.214556 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:46.214663 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 06:45:46.214726 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":106,"threads_started":1}
I20260928 06:45:46.221148 15192 raft_consensus.cc:1261] T 67097c2134cb485883d82af055d8e596 P bd72f0024ada43e7a14d4e1bee99f7e6 [term 1 FOLLOWER]: Refusing update from remote peer f2778a9828114f36ae7f9421329e9dfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:46.221251 15052 raft_consensus.cc:1261] T 67097c2134cb485883d82af055d8e596 P 69f4f371eac34d12bca22a7bda991178 [term 1 FOLLOWER]: Refusing update from remote peer f2778a9828114f36ae7f9421329e9dfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:45:46.221552 15413 consensus_queue.cc:1035] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd72f0024ada43e7a14d4e1bee99f7e6" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:46.221737 15502 consensus_queue.cc:1035] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f4f371eac34d12bca22a7bda991178" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:45:46.239609 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:46.235945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:59025 (local address 127.7.48.3:36477)
0928 06:45:46.236069 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:46.236073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:46.236225 (+ 152us) server_negotiation.cc:407] Connection header received
0928 06:45:46.236405 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:46.236411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:46.236465 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:46.236529 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:45:46.236767 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:46.237248 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:46.238479 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:46.238750 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:46.238867 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:46.239109 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:45:46.239194 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:45:46.239262 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:46.239414 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 06:45:46.239502 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:45:46.239619 15522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:46.235845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:36477 (local address 127.7.48.1:59025)
0928 06:45:46.236145 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:46.236159 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:46.236287 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:46.236554 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:46.236558 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:46.236567 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:45:46.236670 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:46.236676 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:46.237349 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:46.237352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:46.238334 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:46.238343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:46.238867 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:46.238871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:46.238966 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:46.238996 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:45:46.239302 (+ 306us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:45:46.239308 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:45:46.239383 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 06:45:46.239470 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":120,"threads_started":1}
I20260928 06:45:46.257889 14896 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:46.258077 14896 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:45:46.260020 14896 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34b336e939ad476ba98f1ae1063ffdcc: Sending DeleteTablet for 3 replicas of tablet 67097c2134cb485883d82af055d8e596
I20260928 06:45:46.260576 15033 tablet_service.cc:1514] Processing DeleteTablet for tablet 67097c2134cb485883d82af055d8e596 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:46 UTC) from {username='oryx', principal='oryx/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:60448
I20260928 06:45:46.260623 15313 tablet_service.cc:1514] Processing DeleteTablet for tablet 67097c2134cb485883d82af055d8e596 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:46 UTC) from {username='oryx', principal='oryx/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:59870
I20260928 06:45:46.260685 15173 tablet_service.cc:1514] Processing DeleteTablet for tablet 67097c2134cb485883d82af055d8e596 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:45:46 UTC) from {username='oryx', principal='oryx/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:39484
I20260928 06:45:46.260775 15480 tablet_replica.cc:330] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd: stopping tablet replica
I20260928 06:45:46.260797 15481 tablet_replica.cc:330] T 67097c2134cb485883d82af055d8e596 P 69f4f371eac34d12bca22a7bda991178: stopping tablet replica
I20260928 06:45:46.260861 15483 tablet_replica.cc:330] T 67097c2134cb485883d82af055d8e596 P bd72f0024ada43e7a14d4e1bee99f7e6: stopping tablet replica
I20260928 06:45:46.260900 15480 raft_consensus.cc:2229] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:45:46.260923 15481 raft_consensus.cc:2229] T 67097c2134cb485883d82af055d8e596 P 69f4f371eac34d12bca22a7bda991178 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:46.260954 15483 raft_consensus.cc:2229] T 67097c2134cb485883d82af055d8e596 P bd72f0024ada43e7a14d4e1bee99f7e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:45:46.261034 15481 raft_consensus.cc:2258] T 67097c2134cb485883d82af055d8e596 P 69f4f371eac34d12bca22a7bda991178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:46.261040 15483 raft_consensus.cc:2258] T 67097c2134cb485883d82af055d8e596 P bd72f0024ada43e7a14d4e1bee99f7e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:46.261053 15480 raft_consensus.cc:2258] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:45:46.261785 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 14968
I20260928 06:45:46.261976 15483 ts_tablet_manager.cc:1905] T 67097c2134cb485883d82af055d8e596 P bd72f0024ada43e7a14d4e1bee99f7e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:46.262017 15480 ts_tablet_manager.cc:1905] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:45:46.264577 15480 ts_tablet_manager.cc:1918] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:45:46.264577 15483 ts_tablet_manager.cc:1918] T 67097c2134cb485883d82af055d8e596 P bd72f0024ada43e7a14d4e1bee99f7e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:45:46.264662 15483 log.cc:1196] T 67097c2134cb485883d82af055d8e596 P bd72f0024ada43e7a14d4e1bee99f7e6: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/67097c2134cb485883d82af055d8e596
I20260928 06:45:46.264818 15480 log.cc:1196] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/67097c2134cb485883d82af055d8e596
I20260928 06:45:46.264998 15483 ts_tablet_manager.cc:1939] T 67097c2134cb485883d82af055d8e596 P bd72f0024ada43e7a14d4e1bee99f7e6: Deleting consensus metadata
I20260928 06:45:46.265093 15480 ts_tablet_manager.cc:1939] T 67097c2134cb485883d82af055d8e596 P f2778a9828114f36ae7f9421329e9dfd: Deleting consensus metadata
I20260928 06:45:46.265861 14882 catalog_manager.cc:4841] TS f2778a9828114f36ae7f9421329e9dfd (127.7.48.3:36477): tablet 67097c2134cb485883d82af055d8e596 (table test-table [id=0825613668b3455c8996d413f6c05bcb]) successfully deleted
I20260928 06:45:46.265949 14880 catalog_manager.cc:4841] TS bd72f0024ada43e7a14d4e1bee99f7e6 (127.7.48.2:40247): tablet 67097c2134cb485883d82af055d8e596 (table test-table [id=0825613668b3455c8996d413f6c05bcb]) successfully deleted
W20260928 06:45:46.271675 14882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.48.1:36379 (error 108)
I20260928 06:45:46.272063 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 15108
I20260928 06:45:46.273278 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:46.272821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:36379 (local address 127.0.0.1:54568)
0928 06:45:46.273054 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:46.273141 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.1:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":74,"threads_started":1}
W20260928 06:45:46.273627 14882 catalog_manager.cc:4584] TS 69f4f371eac34d12bca22a7bda991178 (127.7.48.1:36379): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67097c2134cb485883d82af055d8e596: Network error: Client connection negotiation failed: client connection to 127.7.48.1:36379: connect: Connection refused (error 111)
I20260928 06:45:46.279745 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 15248
I20260928 06:45:46.286707 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 14864
2026-09-28T06:45:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3975 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:45:46 dist-test-slave-twl6 krb5kdc[15528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:45:46 dist-test-slave-twl6 krb5kdc[15528](info): set up 2 sockets
Sep 28 06:45:46 dist-test-slave-twl6 krb5kdc[15528](info): commencing operation
krb5kdc: starting...
W20260928 06:45:47.321012 7360 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:45:47 dist-test-slave-twl6 krb5kdc[15528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:45:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:45:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:45:47.445230 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:32837
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:43475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:32837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:47.555559 15541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:47.555899 15541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:47.555994 15541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:47.559930 15541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:45:47.560040 15541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:47.560077 15541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:47.560110 15541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:45:47.560141 15541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:45:47.565369 15541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:43475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:32837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:32837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:47.566522 15541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:47.567601 15541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:47.576067 15547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:47.576123 15546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:47.576066 15549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:47.576160 15541 server_base.cc:1034] running on GCE node
I20260928 06:45:47.576874 15541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:47.578001 15541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:47.579200 15541 hybrid_clock.cc:648] HybridClock initialized: now 1790577947579169 us; error 49 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T06:45:47Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1295 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:45:47 dist-test-slave-twl6 krb5kdc[15554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:45:47 dist-test-slave-twl6 krb5kdc[15554](info): set up 2 sockets
Sep 28 06:45:47 dist-test-slave-twl6 krb5kdc[15554](info): commencing operation
krb5kdc: starting...
W20260928 06:45:48.605032 7360 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:45:48 dist-test-slave-twl6 krb5kdc[15554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:45:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:45:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:45:48.730737 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:41517
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:46089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:41517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:48.841956 15567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:48.842267 15567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:48.842307 15567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:48.846249 15567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:45:48.846307 15567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:48.846326 15567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:48.846342 15567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:45:48.846357 15567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:45:48.851554 15567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:46089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:41517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:41517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:48.852785 15567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:48.853847 15567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:48.862737 15575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:48.862759 15572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:48.862747 15573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:48.862995 15567 server_base.cc:1034] running on GCE node
I20260928 06:45:48.863655 15567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:48.864842 15567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:48.866003 15567 hybrid_clock.cc:648] HybridClock initialized: now 1790577948865980 us; error 38 us; skew 500 ppm
Sep 28 06:45:48 dist-test-slave-twl6 krb5kdc[15554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577948, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:48.868880 15567 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:45:48.869803 15567 webserver.cc:460] Webserver started at http://127.7.48.62:39843/ using document root <none> and password file <none>
I20260928 06:45:48.870393 15567 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:48.870487 15567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:48.870728 15567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:48.872882 15567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "7606c77fc2d94505adfd04ec39daa93e"
format_stamp: "Formatted at 2026-09-28 06:45:48 on dist-test-slave-twl6"
server_key: "2c7ee9ed55cf686159be3130a21c317f"
server_key_iv: "228b8e94fe1dd1aaf42b169ccd2c706b"
server_key_version: "encryptionkey@0"
I20260928 06:45:48.873464 15567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "7606c77fc2d94505adfd04ec39daa93e"
format_stamp: "Formatted at 2026-09-28 06:45:48 on dist-test-slave-twl6"
server_key: "2c7ee9ed55cf686159be3130a21c317f"
server_key_iv: "228b8e94fe1dd1aaf42b169ccd2c706b"
server_key_version: "encryptionkey@0"
I20260928 06:45:48.877731 15567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:45:48.880272 15582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:48.881458 15567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:45:48.881594 15567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "7606c77fc2d94505adfd04ec39daa93e"
format_stamp: "Formatted at 2026-09-28 06:45:48 on dist-test-slave-twl6"
server_key: "2c7ee9ed55cf686159be3130a21c317f"
server_key_iv: "228b8e94fe1dd1aaf42b169ccd2c706b"
server_key_version: "encryptionkey@0"
I20260928 06:45:48.881738 15567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:48.901227 15567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:48.904386 15567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:48.904601 15567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:48.911777 15567 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:41517
I20260928 06:45:48.911800 15634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:41517 every 8 connection(s)
I20260928 06:45:48.912777 15567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:45:48.915640 15635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:45:48.917368 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 15567
I20260928 06:45:48.917493 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:45:48.917771 7360 external_mini_cluster.cc:1428] Setting key 0654c3c77fe5424b73941b1a88361b55
I20260928 06:45:48.921877 15635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e: Bootstrap starting.
I20260928 06:45:48.924153 15635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e: Neither blocks nor log segments found. Creating new log.
I20260928 06:45:48.924983 15635 log.cc:824] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e: Log is configured to *not* fsync() on all Append() calls
Sep 28 06:45:48 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:48.927208 15635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e: No bootstrap required, opened a new log
I20260928 06:45:48.930773 15635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7606c77fc2d94505adfd04ec39daa93e" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 41517 } }
I20260928 06:45:48.931123 15635 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:45:48.931209 15635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7606c77fc2d94505adfd04ec39daa93e, State: Initialized, Role: FOLLOWER
I20260928 06:45:48.931679 15635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7606c77fc2d94505adfd04ec39daa93e" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 41517 } }
I20260928 06:45:48.931854 15635 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:45:48.931944 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:48.919221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.7.48.62:41517)
0928 06:45:48.919635 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:48.919650 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:48.919685 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:45:48.920362 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:48.920380 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:48.920707 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:48.921045 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:48.921818 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:48.922818 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:48.923686 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:48.924241 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:48.924428 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:48.927320 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:48.927328 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:48.927345 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:48.927382 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:48.929041 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:48.929496 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:48.929501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:48.929505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:48.929607 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:48.929904 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:48.929912 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:48.929913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:48.930276 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:48.930479 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:48.930755 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 06:45:48.931023 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":117,"threads_started":1}
I20260928 06:45:48.931952 15635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:45:48.932124 15635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:45:48.933075 15635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7606c77fc2d94505adfd04ec39daa93e" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 41517 } }
I20260928 06:45:48.933440 15635 leader_election.cc:302] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7606c77fc2d94505adfd04ec39daa93e; no voters:
I20260928 06:45:48.933776 15635 leader_election.cc:288] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:45:48.933898 15640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:45:48.934142 15640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [term 1 LEADER]: Becoming Leader. State: Replica: 7606c77fc2d94505adfd04ec39daa93e, State: Running, Role: LEADER
I20260928 06:45:48.934501 15640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7606c77fc2d94505adfd04ec39daa93e" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 41517 } }
I20260928 06:45:48.935009 15635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:45:48.935627 15640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7606c77fc2d94505adfd04ec39daa93e. Latest consensus state: current_term: 1 leader_uuid: "7606c77fc2d94505adfd04ec39daa93e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7606c77fc2d94505adfd04ec39daa93e" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 41517 } } }
I20260928 06:45:48.935842 15640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:48.936085 15642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7606c77fc2d94505adfd04ec39daa93e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7606c77fc2d94505adfd04ec39daa93e" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 41517 } } }
I20260928 06:45:48.936180 15642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [sys.catalog]: This master's current role is: LEADER
W20260928 06:45:48.939851 15655 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:45:48.939981 15655 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:45:48.940053 15649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:45:48.940725 15649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:45:48.946192 15649 catalog_manager.cc:1347] Generated new cluster ID: 817fd5a0293d4ce2b255f74742551576
I20260928 06:45:48.946252 15649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:45:48.971284 15649 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:45:48.972672 15649 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:45:48.978243 15649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e: Generated new TSK 0
I20260928 06:45:48.978941 15649 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
I20260928 06:45:49.016147 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:41517
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:46089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:49.131016 15661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:49.131284 15661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:49.131327 15661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:49.135572 15661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:49.135636 15661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:49.135694 15661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:45:49.141198 15661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:46089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:41517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:49.142364 15661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:49.143420 15661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:49.153198 15669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:49.153216 15667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:49.153216 15666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:49.153548 15661 server_base.cc:1034] running on GCE node
I20260928 06:45:49.154021 15661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:49.154695 15661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:49.155925 15661 hybrid_clock.cc:648] HybridClock initialized: now 1790577949155907 us; error 29 us; skew 500 ppm
Sep 28 06:45:49 dist-test-slave-twl6 krb5kdc[15554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577949, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:49.158720 15661 init.cc:377] Logged in from keytab as kudu/127.7.48.1@KRBTEST.COM (short username kudu)
I20260928 06:45:49.159693 15661 webserver.cc:460] Webserver started at http://127.7.48.1:33887/ using document root <none> and password file <none>
I20260928 06:45:49.160367 15661 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:49.160462 15661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:49.160686 15661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:49.162828 15661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "16b14c08259347fcaae95666c43e4d11"
format_stamp: "Formatted at 2026-09-28 06:45:49 on dist-test-slave-twl6"
server_key: "69b2b10a3e0a70c85495e673fb990092"
server_key_iv: "a8cf79582354cebc05a8021e540ee713"
server_key_version: "encryptionkey@0"
I20260928 06:45:49.163424 15661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "16b14c08259347fcaae95666c43e4d11"
format_stamp: "Formatted at 2026-09-28 06:45:49 on dist-test-slave-twl6"
server_key: "69b2b10a3e0a70c85495e673fb990092"
server_key_iv: "a8cf79582354cebc05a8021e540ee713"
server_key_version: "encryptionkey@0"
I20260928 06:45:49.167306 15661 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 06:45:49.169889 15676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:49.171169 15661 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:49.171309 15661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "16b14c08259347fcaae95666c43e4d11"
format_stamp: "Formatted at 2026-09-28 06:45:49 on dist-test-slave-twl6"
server_key: "69b2b10a3e0a70c85495e673fb990092"
server_key_iv: "a8cf79582354cebc05a8021e540ee713"
server_key_version: "encryptionkey@0"
I20260928 06:45:49.171425 15661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:49.193832 15661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:49.197082 15661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:49.197297 15661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:49.197960 15661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:49.198814 15661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:49.198886 15661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:49.198953 15661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:49.199003 15661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:49.209049 15661 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:45727
I20260928 06:45:49.209077 15789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:45727 every 8 connection(s)
I20260928 06:45:49.209975 15661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:45:49.212601 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 15661
I20260928 06:45:49.212747 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
I20260928 06:45:49.213056 7360 external_mini_cluster.cc:1428] Setting key 43989b2014205ae27ebfcc59d1b32ab8
Sep 28 06:45:49 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577949, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.48.2@KRBTEST.COM; defaulting to no policy
I20260928 06:45:49.225390 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:49.211769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:46147 (local address 127.7.48.62:41517)
0928 06:45:49.211918 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:49.211923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:49.213171 (+ 1248us) server_negotiation.cc:407] Connection header received
0928 06:45:49.214300 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:49.214303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:49.214352 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:49.214467 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:49.216130 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:49.216690 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:49.218312 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:49.218747 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:49.218871 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:49.221629 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:49.221633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:49.221640 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:49.221665 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:49.223230 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:49.223742 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:49.223746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:49.223748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:49.223825 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:49.224256 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:49.224259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:49.224261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:49.224446 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:49.224555 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:49.225065 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 06:45:49.225216 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.2@KRBTEST.COM" created.
I20260928 06:45:49.226559 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:49.212178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.1:46147)
0928 06:45:49.212981 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:49.213039 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:49.214096 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:49.214672 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:49.214682 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:49.215212 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:49.215963 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:49.215987 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:49.216822 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:49.216828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:49.218173 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:49.218183 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:49.218893 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:49.218895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:49.219033 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:49.219673 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:49.219690 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:49.221454 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:49.223354 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:49.223360 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:49.223376 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:49.223633 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:49.223951 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:49.223954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:49.223955 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:49.224117 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:49.224554 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:49.224561 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:49.224818 (+ 257us) client_negotiation.cc:769] Sending connection context
0928 06:45:49.225012 (+ 194us) client_negotiation.cc:240] Negotiation successful
0928 06:45:49.225427 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":111,"threads_started":1}
I20260928 06:45:49.227778 15790 heartbeater.cc:347] Connected to a master server at 127.7.48.62:41517
I20260928 06:45:49.228055 15790 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:49.228559 15790 heartbeater.cc:511] Master 127.7.48.62:41517 requested a full tablet report, sending...
I20260928 06:45:49.230208 15599 ts_manager.cc:194] Registered new tserver with Master: 16b14c08259347fcaae95666c43e4d11 (127.7.48.1:45727)
I20260928 06:45:49.232002 15599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.1@KRBTEST.COM'} at 127.7.48.1:46147
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
I20260928 06:45:49.242871 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:41517
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:46089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:49.356482 15795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:49.356786 15795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:49.356889 15795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:49.361037 15795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:49.361145 15795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:49.361240 15795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:45:49.366596 15795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:46089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:41517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:49.367712 15795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:49.368886 15795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:49.378908 15801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:49.378942 15800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:49.378908 15803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:49.379117 15795 server_base.cc:1034] running on GCE node
I20260928 06:45:49.379707 15795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:49.380409 15795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:49.381588 15795 hybrid_clock.cc:648] HybridClock initialized: now 1790577949381570 us; error 26 us; skew 500 ppm
Sep 28 06:45:49 dist-test-slave-twl6 krb5kdc[15554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577949, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:49.384528 15795 init.cc:377] Logged in from keytab as kudu/127.7.48.2@KRBTEST.COM (short username kudu)
I20260928 06:45:49.385387 15795 webserver.cc:460] Webserver started at http://127.7.48.2:42973/ using document root <none> and password file <none>
I20260928 06:45:49.386009 15795 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:49.386071 15795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:49.386313 15795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:49.388535 15795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "01250be8819847d294d402ef20a35b0e"
format_stamp: "Formatted at 2026-09-28 06:45:49 on dist-test-slave-twl6"
server_key: "1e937c006019e90c58c06e5ebc8065cc"
server_key_iv: "6df2b4e420a108e589c55adee19ecd95"
server_key_version: "encryptionkey@0"
I20260928 06:45:49.389099 15795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "01250be8819847d294d402ef20a35b0e"
format_stamp: "Formatted at 2026-09-28 06:45:49 on dist-test-slave-twl6"
server_key: "1e937c006019e90c58c06e5ebc8065cc"
server_key_iv: "6df2b4e420a108e589c55adee19ecd95"
server_key_version: "encryptionkey@0"
I20260928 06:45:49.392903 15795 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 06:45:49.395355 15810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:49.396653 15795 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 06:45:49.396773 15795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "01250be8819847d294d402ef20a35b0e"
format_stamp: "Formatted at 2026-09-28 06:45:49 on dist-test-slave-twl6"
server_key: "1e937c006019e90c58c06e5ebc8065cc"
server_key_iv: "6df2b4e420a108e589c55adee19ecd95"
server_key_version: "encryptionkey@0"
I20260928 06:45:49.396875 15795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:49.412482 15795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:49.415591 15795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:49.415800 15795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:49.416426 15795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:49.417286 15795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:49.417346 15795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:49.417448 15795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:49.417486 15795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:49.426426 15795 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:35763
I20260928 06:45:49.426450 15923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:35763 every 8 connection(s)
I20260928 06:45:49.427325 15795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:45:49.429085 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 15795
I20260928 06:45:49.429216 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:45:49.429523 7360 external_mini_cluster.cc:1428] Setting key 34b9562a4a33c32672ea447496aa4fe6
Sep 28 06:45:49 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577949, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.48.3@KRBTEST.COM; defaulting to no policy
I20260928 06:45:49.442373 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:49.429222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:43665 (local address 127.7.48.62:41517)
0928 06:45:49.429385 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:49.429389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:49.430762 (+ 1373us) server_negotiation.cc:407] Connection header received
0928 06:45:49.431734 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:49.431738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:49.431787 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:49.431885 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:49.433048 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:49.433583 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:49.435418 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:49.435644 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:49.435730 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:49.438693 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:49.438697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:49.438699 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:49.438720 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:49.440302 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:49.440816 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:49.440819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:49.440826 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:49.440900 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:49.441257 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:49.441261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:49.441263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:49.441463 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:49.441574 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:49.442078 (+ 504us) server_negotiation.cc:299] Negotiation successful
0928 06:45:49.442207 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.3@KRBTEST.COM" created.
I20260928 06:45:49.443430 15926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:49.429785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.2:43665)
0928 06:45:49.430594 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:49.430652 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:49.431525 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:49.432088 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:49.432094 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:49.432444 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:49.432906 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:49.432919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:49.433719 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:49.433725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:49.435294 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:49.435309 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:49.435817 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:49.435822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:49.436104 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:49.436801 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:49.436826 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:49.438517 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:49.440422 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:49.440428 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:49.440443 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:49.440701 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:49.440994 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:49.440998 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:49.441000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:49.441148 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:49.441572 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:49.441579 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:49.441841 (+ 262us) client_negotiation.cc:769] Sending connection context
0928 06:45:49.442038 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 06:45:49.442384 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":133,"threads_started":1}
I20260928 06:45:49.444761 15924 heartbeater.cc:347] Connected to a master server at 127.7.48.62:41517
I20260928 06:45:49.445044 15924 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:49.445675 15924 heartbeater.cc:511] Master 127.7.48.62:41517 requested a full tablet report, sending...
I20260928 06:45:49.446731 15599 ts_manager.cc:194] Registered new tserver with Master: 01250be8819847d294d402ef20a35b0e (127.7.48.2:35763)
I20260928 06:45:49.447342 15599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.2@KRBTEST.COM'} at 127.7.48.2:43665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
I20260928 06:45:49.459508 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:41517
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:46089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:49.571782 15929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:49.572100 15929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:49.572145 15929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:49.576287 15929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:49.576340 15929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:49.576398 15929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:45:49.581676 15929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:46089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:41517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:49.582700 15929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:49.583676 15929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:49.593726 15934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:49.593737 15937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:49.593796 15935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:49.593992 15929 server_base.cc:1034] running on GCE node
I20260928 06:45:49.594408 15929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:49.595032 15929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:49.596215 15929 hybrid_clock.cc:648] HybridClock initialized: now 1790577949596196 us; error 30 us; skew 500 ppm
Sep 28 06:45:49 dist-test-slave-twl6 krb5kdc[15554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577949, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:49.598999 15929 init.cc:377] Logged in from keytab as kudu/127.7.48.3@KRBTEST.COM (short username kudu)
I20260928 06:45:49.599924 15929 webserver.cc:460] Webserver started at http://127.7.48.3:44705/ using document root <none> and password file <none>
I20260928 06:45:49.600540 15929 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:49.600628 15929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:49.600916 15929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:45:49.603062 15929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "519867388f5649d5987523a8a11e9573"
format_stamp: "Formatted at 2026-09-28 06:45:49 on dist-test-slave-twl6"
server_key: "3b223924999adb6780d40211cfa8d838"
server_key_iv: "2a03d0bf4e6458874960ea9454ea0c89"
server_key_version: "encryptionkey@0"
I20260928 06:45:49.603619 15929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "519867388f5649d5987523a8a11e9573"
format_stamp: "Formatted at 2026-09-28 06:45:49 on dist-test-slave-twl6"
server_key: "3b223924999adb6780d40211cfa8d838"
server_key_iv: "2a03d0bf4e6458874960ea9454ea0c89"
server_key_version: "encryptionkey@0"
I20260928 06:45:49.607488 15929 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 06:45:49.609957 15944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:49.611119 15929 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 06:45:49.611260 15929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "519867388f5649d5987523a8a11e9573"
format_stamp: "Formatted at 2026-09-28 06:45:49 on dist-test-slave-twl6"
server_key: "3b223924999adb6780d40211cfa8d838"
server_key_iv: "2a03d0bf4e6458874960ea9454ea0c89"
server_key_version: "encryptionkey@0"
I20260928 06:45:49.611371 15929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:49.631855 15929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:49.635474 15929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:49.635696 15929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:49.636373 15929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:49.637235 15929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:49.637281 15929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:49.637321 15929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:49.637343 15929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:49.646241 15929 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:39885
I20260928 06:45:49.646276 16057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:39885 every 8 connection(s)
I20260928 06:45:49.647163 15929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:45:49 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577949, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:45:49.656330 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 15929
I20260928 06:45:49.656474 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:45:49.656771 7360 external_mini_cluster.cc:1428] Setting key 1108130eb3b0f14daafe283be582f212
I20260928 06:45:49.660632 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:49.648997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:44165 (local address 127.7.48.62:41517)
0928 06:45:49.649130 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:49.649134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:49.650172 (+ 1038us) server_negotiation.cc:407] Connection header received
0928 06:45:49.650983 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:49.650985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:49.651032 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:49.651111 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:49.652324 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:49.652852 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:49.653951 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:49.654160 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:49.654235 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:49.656846 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:49.656850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:49.656854 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:49.656881 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:49.658443 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:49.659092 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:49.659096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:49.659098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:49.659208 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:49.659598 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:49.659602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:49.659603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:49.659804 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:49.659936 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:49.660407 (+ 471us) server_negotiation.cc:299] Negotiation successful
0928 06:45:49.660517 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:45:49.661499 16060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:49.649373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.3:44165)
0928 06:45:49.650026 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:49.650066 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:49.650811 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:49.651237 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:49.651244 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:49.651612 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:49.652177 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:49.652191 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:49.652961 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:49.652963 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:49.653846 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:49.653854 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:49.654283 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:49.654285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:49.654419 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:49.655035 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:49.655052 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:49.656674 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:45:49.658598 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:49.658606 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:49.658620 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:49.658970 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:49.659300 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:45:49.659303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:45:49.659305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:45:49.659487 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:45:49.659924 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:45:49.659930 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:45:49.660202 (+ 272us) client_negotiation.cc:769] Sending connection context
0928 06:45:49.660361 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 06:45:49.660620 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":116,"threads_started":1}
I20260928 06:45:49.662532 16058 heartbeater.cc:347] Connected to a master server at 127.7.48.62:41517
I20260928 06:45:49.662784 16058 heartbeater.cc:464] Registering TS with master...
I20260928 06:45:49.663272 16058 heartbeater.cc:511] Master 127.7.48.62:41517 requested a full tablet report, sending...
I20260928 06:45:49.664299 15599 ts_manager.cc:194] Registered new tserver with Master: 519867388f5649d5987523a8a11e9573 (127.7.48.3:39885)
I20260928 06:45:49.664892 15599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.3@KRBTEST.COM'} at 127.7.48.3:44165
I20260928 06:45:49.670553 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.7.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.1.keytab.
Entry for principal oryx/127.7.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.1.keytab.
WARNING: no policy specified for oryx/127.7.48.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.48.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.2.keytab.
Entry for principal oryx/127.7.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.2.keytab.
WARNING: no policy specified for oryx/127.7.48.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.48.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.3.keytab.
Entry for principal oryx/127.7.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.3.keytab.
I20260928 06:45:49.743093 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 15661
I20260928 06:45:49.749239 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 15795
I20260928 06:45:49.755467 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 15929
I20260928 06:45:49.760823 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 15567
I20260928 06:45:49.767119 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:41517
--webserver_interface=127.7.48.62
--webserver_port=39843
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:46089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:41517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:45:49.876302 16067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:49.876569 16067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:49.876646 16067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:49.880533 16067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:45:49.880589 16067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:49.880609 16067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:49.880625 16067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:45:49.880641 16067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:45:49.885794 16067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:46089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:41517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:41517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=39843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:49.886884 16067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:49.888001 16067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:49.896412 16075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:49.896431 16073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:49.896478 16072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:49.896797 16067 server_base.cc:1034] running on GCE node
I20260928 06:45:49.897364 16067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:49.898403 16067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:49.899561 16067 hybrid_clock.cc:648] HybridClock initialized: now 1790577949899531 us; error 43 us; skew 500 ppm
Sep 28 06:45:49 dist-test-slave-twl6 krb5kdc[15554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577949, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:49.902318 16067 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:45:49.903288 16067 webserver.cc:460] Webserver started at http://127.7.48.62:39843/ using document root <none> and password file <none>
I20260928 06:45:49.903848 16067 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:49.903983 16067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:49.907701 16067 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:49.909832 16082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:49.911017 16067 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:45:49.911170 16067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "7606c77fc2d94505adfd04ec39daa93e"
format_stamp: "Formatted at 2026-09-28 06:45:48 on dist-test-slave-twl6"
server_key: "2c7ee9ed55cf686159be3130a21c317f"
server_key_iv: "228b8e94fe1dd1aaf42b169ccd2c706b"
server_key_version: "encryptionkey@0"
I20260928 06:45:49.911705 16067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:49.928459 16067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:49.931354 16067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:49.931571 16067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:49.938759 16067 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:41517
I20260928 06:45:49.938769 16134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:41517 every 8 connection(s)
I20260928 06:45:49.939813 16067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:45:49.943728 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 16067
I20260928 06:45:49.944212 16135 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:45:49.944258 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:45727
--local_ip_for_outbound_sockets=127.7.48.1
--tserver_master_addrs=127.7.48.62:41517
--webserver_port=33887
--webserver_interface=127.7.48.1
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:46089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.48.1
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:45:49.946592 16135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e: Bootstrap starting.
I20260928 06:45:49.963294 16135 log.cc:824] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:45:49.970055 16135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:45:49.970470 16135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e: Bootstrap complete.
I20260928 06:45:49.974725 16135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7606c77fc2d94505adfd04ec39daa93e" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 41517 } }
I20260928 06:45:49.975127 16135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7606c77fc2d94505adfd04ec39daa93e, State: Initialized, Role: FOLLOWER
I20260928 06:45:49.975629 16135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7606c77fc2d94505adfd04ec39daa93e" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 41517 } }
I20260928 06:45:49.975746 16135 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:45:49.975835 16135 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:45:49.975955 16135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:45:49.977875 16135 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7606c77fc2d94505adfd04ec39daa93e" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 41517 } }
I20260928 06:45:49.978217 16135 leader_election.cc:302] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7606c77fc2d94505adfd04ec39daa93e; no voters:
I20260928 06:45:49.978552 16135 leader_election.cc:288] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:45:49.978660 16139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:45:49.978847 16139 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [term 2 LEADER]: Becoming Leader. State: Replica: 7606c77fc2d94505adfd04ec39daa93e, State: Running, Role: LEADER
I20260928 06:45:49.979156 16139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7606c77fc2d94505adfd04ec39daa93e" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 41517 } }
I20260928 06:45:49.979852 16135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:45:49.980532 16140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7606c77fc2d94505adfd04ec39daa93e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7606c77fc2d94505adfd04ec39daa93e" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 41517 } } }
I20260928 06:45:49.980683 16140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:49.981016 16141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7606c77fc2d94505adfd04ec39daa93e. Latest consensus state: current_term: 2 leader_uuid: "7606c77fc2d94505adfd04ec39daa93e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7606c77fc2d94505adfd04ec39daa93e" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 41517 } } }
I20260928 06:45:49.981607 16141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e [sys.catalog]: This master's current role is: LEADER
I20260928 06:45:49.981691 16146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:45:49.984763 16146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:45:49.985391 16146 catalog_manager.cc:1259] Loaded cluster ID: 817fd5a0293d4ce2b255f74742551576
I20260928 06:45:49.985453 16146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:45:49.987787 16146 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:45:49.988453 16146 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7606c77fc2d94505adfd04ec39daa93e: Loaded TSK: 0
I20260928 06:45:49.989674 16146 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:45:50.060335 16137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:50.060636 16137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:50.060711 16137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:50.064805 16137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:50.064886 16137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:50.064970 16137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:45:50.070363 16137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:46089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.1.keytab
--principal=oryx/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:45727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=33887
--tserver_master_addrs=127.7.48.62:41517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:50.071412 16137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:50.072502 16137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:50.083278 16161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:50.083546 16137 server_base.cc:1034] running on GCE node
W20260928 06:45:50.083294 16164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:50.083339 16162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:50.084230 16137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:50.084887 16137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:50.086091 16137 hybrid_clock.cc:648] HybridClock initialized: now 1790577950086069 us; error 49 us; skew 500 ppm
Sep 28 06:45:50 dist-test-slave-twl6 krb5kdc[15554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577950, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:50.094877 16137 init.cc:377] Logged in from keytab as oryx/127.7.48.1@KRBTEST.COM (short username oryx)
I20260928 06:45:50.095984 16137 webserver.cc:460] Webserver started at http://127.7.48.1:33887/ using document root <none> and password file <none>
I20260928 06:45:50.096588 16137 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:50.096706 16137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:50.100909 16137 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:45:50.104128 16171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:50.105598 16137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:45:50.105738 16137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "16b14c08259347fcaae95666c43e4d11"
format_stamp: "Formatted at 2026-09-28 06:45:49 on dist-test-slave-twl6"
server_key: "69b2b10a3e0a70c85495e673fb990092"
server_key_iv: "a8cf79582354cebc05a8021e540ee713"
server_key_version: "encryptionkey@0"
I20260928 06:45:50.106509 16137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:50.140035 16137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:50.144479 16137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:50.144734 16137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:50.145624 16137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:50.146927 16137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:50.147002 16137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:50.147097 16137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:50.147130 16137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:50.156725 16137 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:45727
I20260928 06:45:50.156769 16284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:45727 every 8 connection(s)
I20260928 06:45:50.157859 16137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:45:50.161476 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 16137
I20260928 06:45:50.161922 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:35763
--local_ip_for_outbound_sockets=127.7.48.2
--tserver_master_addrs=127.7.48.62:41517
--webserver_port=42973
--webserver_interface=127.7.48.2
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:46089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.48.2
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:45:50 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:50 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:50.172832 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:50.160159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.1:56851)
0928 06:45:50.160872 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:50.160923 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:50.161803 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:50.163230 (+ 1427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:50.163237 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:50.163824 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:50.164570 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:50.164594 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.166656 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.166658 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:50.167541 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:50.167550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.168776 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.168779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:50.168918 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:50.169571 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:50.169588 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:50.172060 (+ 2472us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":137,"threads_started":1}
W20260928 06:45:50.173424 16287 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:50.160074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:56851 (local address 127.7.48.62:41517)
0928 06:45:50.160535 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:50.160555 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:50.161049 (+ 494us) server_negotiation.cc:407] Connection header received
0928 06:45:50.162290 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:50.162308 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:50.162761 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:50.163148 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:50.164775 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.166470 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.167681 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.168607 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.168823 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:50.172392 (+ 3569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.1:56851: BlockingRecv error: recv got EOF from 127.7.48.1:56851 (error 108)
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":145,"threads_started":1}
W20260928 06:45:50.173489 16285 heartbeater.cc:644] Failed to heartbeat to 127.7.48.62:41517 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.48.62:41517: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Sep 28 06:45:50 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:50 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:50.179339 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:50.174101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.1:45573)
0928 06:45:50.174209 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:50.174223 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:50.174302 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:50.174532 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:50.174535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:50.174670 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:50.174757 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:50.174762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.175771 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.175773 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:50.176367 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:50.176377 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.176976 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.176979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:50.177047 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:50.177548 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:50.177560 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:50.179246 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 06:45:50.179595 16287 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:50.174186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:45573 (local address 127.7.48.62:41517)
0928 06:45:50.174291 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:50.174295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:50.174310 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:45:50.174377 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:50.174380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:50.174424 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:50.174510 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:50.174864 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.175667 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.176508 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.176865 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.176968 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:50.179449 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.1:45573: BlockingRecv error: recv got EOF from 127.7.48.1:45573 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 06:45:50 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:50 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:50.185828 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:50.180109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.1:58449)
0928 06:45:50.180303 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:50.180319 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:50.180396 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:50.180696 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:50.180699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:50.180863 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:50.180951 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:50.180957 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.181756 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.181759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:50.182645 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:50.182656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.183199 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.183202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:50.183289 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:50.183820 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:50.183833 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:50.185687 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 06:45:50.186017 16285 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:45:50.185920 16287 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:50.180226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:58449 (local address 127.7.48.62:41517)
0928 06:45:50.180336 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:50.180339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:50.180361 (+ 22us) server_negotiation.cc:407] Connection header received
0928 06:45:50.180511 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:50.180514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:50.180560 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:50.180669 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:50.181057 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.181660 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.182765 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.183085 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.183183 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:50.185808 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.1:58449: BlockingRecv error: recv got EOF from 127.7.48.1:58449 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 06:45:50.277768 16289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:50.278090 16289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:50.278174 16289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:50.282436 16289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:50.282526 16289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:50.282620 16289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:45:50.287927 16289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:46089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.2.keytab
--principal=oryx/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:35763
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=42973
--tserver_master_addrs=127.7.48.62:41517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:50.289117 16289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:50.290174 16289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:50.299641 16297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:50.299641 16295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:50.299678 16294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:50.300048 16289 server_base.cc:1034] running on GCE node
I20260928 06:45:50.300473 16289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:50.301069 16289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:50.302263 16289 hybrid_clock.cc:648] HybridClock initialized: now 1790577950302253 us; error 30 us; skew 500 ppm
Sep 28 06:45:50 dist-test-slave-twl6 krb5kdc[15554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577950, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:50.305184 16289 init.cc:377] Logged in from keytab as oryx/127.7.48.2@KRBTEST.COM (short username oryx)
I20260928 06:45:50.306022 16289 webserver.cc:460] Webserver started at http://127.7.48.2:42973/ using document root <none> and password file <none>
I20260928 06:45:50.306530 16289 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:50.306599 16289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:50.311128 16289 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:45:50.313506 16304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:50.314791 16289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:45:50.314909 16289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "01250be8819847d294d402ef20a35b0e"
format_stamp: "Formatted at 2026-09-28 06:45:49 on dist-test-slave-twl6"
server_key: "1e937c006019e90c58c06e5ebc8065cc"
server_key_iv: "6df2b4e420a108e589c55adee19ecd95"
server_key_version: "encryptionkey@0"
I20260928 06:45:50.315537 16289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:50.346652 16289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:50.349777 16289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:50.349987 16289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:50.350636 16289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:50.351470 16289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:50.351514 16289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:50.351583 16289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:50.351617 16289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:50.361274 16289 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:35763
I20260928 06:45:50.361301 16417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:35763 every 8 connection(s)
I20260928 06:45:50.362298 16289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:45:50.369933 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 16289
I20260928 06:45:50.370474 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:39885
--local_ip_for_outbound_sockets=127.7.48.3
--tserver_master_addrs=127.7.48.62:41517
--webserver_port=44705
--webserver_interface=127.7.48.3
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:46089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.48.3
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:45:50 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:50 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:45:50.373209 16287 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:50.364117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:43245 (local address 127.7.48.62:41517)
0928 06:45:50.364292 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:50.364296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:50.365409 (+ 1113us) server_negotiation.cc:407] Connection header received
0928 06:45:50.366246 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:50.366249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:50.366299 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:50.366374 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:50.367511 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.368100 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.369164 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.369366 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.369442 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:50.373087 (+ 3645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.2:43245: BlockingRecv error: recv got EOF from 127.7.48.2:43245 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 06:45:50.373849 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:50.364557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.2:43245)
0928 06:45:50.365246 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:50.365287 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:50.366060 (+ 773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:50.366548 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:50.366555 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:50.366910 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:50.367380 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:50.367393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.368210 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.368212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:50.369062 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:50.369070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.369491 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.369493 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:50.369695 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:50.370358 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:50.370384 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:50.372966 (+ 2582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":151,"threads_started":1}
W20260928 06:45:50.374295 16418 heartbeater.cc:644] Failed to heartbeat to 127.7.48.62:41517 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.48.62:41517: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Sep 28 06:45:50 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:50 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:50.379911 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:50.374792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.2:58553)
0928 06:45:50.374899 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:50.374914 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:50.374995 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:50.375259 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:50.375262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:50.375412 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:50.375501 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:50.375506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.376303 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.376309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:50.376846 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:50.376854 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.377296 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.377299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:50.377391 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:50.377886 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:50.377897 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:50.379724 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 06:45:50.380018 16287 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:50.374866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:58553 (local address 127.7.48.62:41517)
0928 06:45:50.374977 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:50.374981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:50.374993 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:45:50.375109 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:50.375112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:50.375158 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:50.375243 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:50.375613 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.376200 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.376971 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.377188 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.377297 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:50.379857 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.2:58553: BlockingRecv error: recv got EOF from 127.7.48.2:58553 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 06:45:50 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:50 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:50.385519 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:50.380437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.2:39663)
0928 06:45:50.380546 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:50.380558 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:50.380641 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:50.381092 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:50.381094 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:50.381216 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:50.381282 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:50.381287 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.382260 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.382262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:50.382739 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:50.382746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.383275 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.383277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:50.383329 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:50.383797 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:50.383808 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:50.385437 (+ 1629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 06:45:50.385680 16287 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:50.380711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:39663 (local address 127.7.48.62:41517)
0928 06:45:50.380869 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:50.380875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:50.380890 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:45:50.380930 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:50.380932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:50.380983 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:50.381067 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:50.381392 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.382160 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.382842 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.383176 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.383272 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:50.385573 (+ 2301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.2:39663: BlockingRecv error: recv got EOF from 127.7.48.2:39663 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 06:45:50.385915 16418 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:45:50.484349 16421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:45:50.484618 16421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:45:50.484663 16421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:45:50.489279 16421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:45:50.489338 16421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:45:50.489399 16421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:45:50.494692 16421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:46089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/krb5kdc/oryx_127.7.48.3.keytab
--principal=oryx/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:39885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=44705
--tserver_master_addrs=127.7.48.62:41517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:45:50.495683 16421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:45:50.496886 16421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:45:50.506531 16426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:45:50.506525 16429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:50.506825 16421 server_base.cc:1034] running on GCE node
W20260928 06:45:50.506861 16427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:45:50.507302 16421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:45:50.507982 16421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:45:50.509145 16421 hybrid_clock.cc:648] HybridClock initialized: now 1790577950509135 us; error 33 us; skew 500 ppm
Sep 28 06:45:50 dist-test-slave-twl6 krb5kdc[15554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577950, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:45:50.515231 16421 init.cc:377] Logged in from keytab as oryx/127.7.48.3@KRBTEST.COM (short username oryx)
I20260928 06:45:50.516268 16421 webserver.cc:460] Webserver started at http://127.7.48.3:44705/ using document root <none> and password file <none>
I20260928 06:45:50.517062 16421 fs_manager.cc:359] Metadata directory not provided
I20260928 06:45:50.517184 16421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:45:50.522823 16421 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:45:50.525252 16436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:50.526489 16421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:45:50.526582 16421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "519867388f5649d5987523a8a11e9573"
format_stamp: "Formatted at 2026-09-28 06:45:49 on dist-test-slave-twl6"
server_key: "3b223924999adb6780d40211cfa8d838"
server_key_iv: "2a03d0bf4e6458874960ea9454ea0c89"
server_key_version: "encryptionkey@0"
I20260928 06:45:50.527153 16421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:45:50.549911 16421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:45:50.553736 16421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:45:50.553972 16421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:45:50.554641 16421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:45:50.555502 16421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:45:50.555549 16421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:50.555622 16421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:45:50.555655 16421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:45:50.564637 16421 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:39885
I20260928 06:45:50.564662 16549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:39885 every 8 connection(s)
I20260928 06:45:50.565656 16421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:45:50.567366 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 16421
Sep 28 06:45:50 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:50 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:50.577750 16554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:50.568407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48872 (local address 127.7.48.62:41517)
0928 06:45:50.568654 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:50.568660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:50.568681 (+ 21us) server_negotiation.cc:407] Connection header received
0928 06:45:50.568751 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:50.568754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:50.568804 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:50.568909 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:50.569596 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.570344 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.571190 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.571532 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.571627 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:50.573202 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:45:50.573209 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:45:50.573226 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:45:50.573256 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:45:50.575435 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:50.575897 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:50.575904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:50.575911 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:50.576073 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:45:50.576371 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:45:50.576373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:45:50.576375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:45:50.576868 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:45:50.577059 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 06:45:50.577354 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 06:45:50.577619 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":97,"threads_started":1}
W20260928 06:45:50.577874 16287 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:50.567512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:48983 (local address 127.7.48.62:41517)
0928 06:45:50.567676 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:50.567682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:50.569208 (+ 1526us) server_negotiation.cc:407] Connection header received
0928 06:45:50.570234 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:50.570237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:50.570274 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:50.570333 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:50.571661 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.572393 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.573790 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.574042 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.574125 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:50.577771 (+ 3646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.3:48983: BlockingRecv error: recv got EOF from 127.7.48.3:48983 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 06:45:50.578436 16553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:50.568001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.3:48983)
0928 06:45:50.569069 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:50.569116 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:50.570020 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:50.570515 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:50.570522 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:50.570840 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:50.571432 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:50.571450 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.572525 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.572528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:50.573663 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:50.573678 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.574201 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.574205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:50.574432 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:50.574917 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:50.574937 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:50.577590 (+ 2653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":148,"threads_started":1}
W20260928 06:45:50.578984 16550 heartbeater.cc:644] Failed to heartbeat to 127.7.48.62:41517 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.48.62:41517: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Sep 28 06:45:50 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:50 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:50.584484 16553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:50.579422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.3:50869)
0928 06:45:50.579560 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:50.579577 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:50.579669 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:50.579960 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:50.579963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:50.580101 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:50.580221 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:50.580228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.580931 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.580937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:50.581591 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:50.581599 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.582032 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.582036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:50.582119 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:50.582668 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:50.582686 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:50.584349 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 06:45:50.584573 16554 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:50.579540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:50869 (local address 127.7.48.62:41517)
0928 06:45:50.579640 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:50.579643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:50.579654 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:45:50.579774 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:50.579776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:50.579820 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:50.579933 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:50.580351 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.580828 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.581706 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.581925 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.582030 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:50.584460 (+ 2430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.3:50869: BlockingRecv error: recv got EOF from 127.7.48.3:50869 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 06:45:50 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:50 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:50.589359 16553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:50.585044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.3:45579)
0928 06:45:50.585158 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:50.585174 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:50.585250 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:50.585433 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:50.585435 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:50.585547 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:50.585610 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:50.585614 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.586259 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.586261 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:50.586859 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:50.586866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.587236 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.587238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:50.587292 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:50.587738 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:50.587749 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:50.589272 (+ 1523us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 06:45:50.589532 16554 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:50.585122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:45579 (local address 127.7.48.62:41517)
0928 06:45:50.585271 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:50.585274 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:50.585285 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:45:50.585313 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:50.585315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:50.585344 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:50.585407 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:50.585710 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.586163 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.586956 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:50.587148 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:50.587218 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:50.589386 (+ 2168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.3:45579: BlockingRecv error: recv got EOF from 127.7.48.3:45579 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260928 06:45:50.589569 16550 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 06:45:51 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:51 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:51.192114 16555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:51.186701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.1:53145)
0928 06:45:51.186993 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:51.187007 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:51.187114 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:51.187331 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:51.187333 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:51.187510 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:51.187611 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:51.187617 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:51.188387 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:51.188389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:51.188985 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:51.188995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:51.189429 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:51.189431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:51.189494 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:51.189989 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:51.190001 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:51.191975 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":114,"threads_started":1}
W20260928 06:45:51.192201 16556 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:51.186750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:53145 (local address 127.7.48.62:41517)
0928 06:45:51.187015 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:51.187018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:51.187070 (+ 52us) server_negotiation.cc:407] Connection header received
0928 06:45:51.187195 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:51.187198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:51.187242 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:51.187320 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:51.187708 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:51.188255 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:51.189104 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:51.189328 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:51.189401 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:51.192089 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.1:53145: BlockingRecv error: recv got EOF from 127.7.48.1:53145 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":116,"threads_started":1}
Sep 28 06:45:51 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:51 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:51.391624 16557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:51.386503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.2:59923)
0928 06:45:51.386754 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:51.386768 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:51.386867 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:51.387155 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:51.387157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:51.387336 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:51.387460 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:51.387465 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:51.388186 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:51.388188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:51.388710 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:51.388717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:51.389113 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:51.389115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:51.389180 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:51.389674 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:51.389685 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:51.391507 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":66,"threads_started":1}
W20260928 06:45:51.391889 16556 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:51.386611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:59923 (local address 127.7.48.62:41517)
0928 06:45:51.386778 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:51.386781 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:51.386835 (+ 54us) server_negotiation.cc:407] Connection header received
0928 06:45:51.386988 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:51.386991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:51.387039 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:51.387115 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:51.387572 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:51.388099 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:51.388811 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:51.389011 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:51.389084 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:51.391644 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.2:59923: BlockingRecv error: recv got EOF from 127.7.48.2:59923 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 06:45:51 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:51 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:51.596158 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:51.590484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.3:40313)
0928 06:45:51.590793 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:51.590805 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:51.590906 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:51.591138 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:51.591140 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:51.591376 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:51.591509 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:51.591516 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:51.592379 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:51.592385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:51.592978 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:51.592987 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:51.593473 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:51.593477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:51.593567 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:51.594075 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:51.594089 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:51.595969 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":88,"threads_started":1}
W20260928 06:45:51.596215 16556 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:51.590558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:40313 (local address 127.7.48.62:41517)
0928 06:45:51.590719 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:51.590722 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:51.590877 (+ 155us) server_negotiation.cc:407] Connection header received
0928 06:45:51.590977 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:51.590979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:51.591026 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:51.591097 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:51.591654 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:51.592261 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:51.593113 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:51.593338 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:51.593469 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:51.596098 (+ 2629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.3:40313: BlockingRecv error: recv got EOF from 127.7.48.3:40313 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 28 06:45:52 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:52 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:52.198597 16559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:52.193047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.1:36551)
0928 06:45:52.193353 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:52.193366 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:52.193491 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:52.193735 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:52.193737 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:52.193921 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:52.194020 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:52.194026 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:52.194776 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:52.194778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:52.195338 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:52.195346 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:52.195782 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:52.195785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:52.195848 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:52.196410 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:52.196421 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:52.198442 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":89,"threads_started":1}
W20260928 06:45:52.198760 16560 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:52.193171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:36551 (local address 127.7.48.62:41517)
0928 06:45:52.193408 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:52.193411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:52.193442 (+ 31us) server_negotiation.cc:407] Connection header received
0928 06:45:52.193566 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:52.193568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:52.193612 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:52.193691 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:52.194137 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:52.194675 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:52.195442 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:52.195663 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:52.195737 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:52.198590 (+ 2853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.1:36551: BlockingRecv error: recv got EOF from 127.7.48.1:36551 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":60,"threads_started":1}
Sep 28 06:45:52 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:52 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:52.398070 16561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:52.392424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.2:34861)
0928 06:45:52.392728 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:52.392743 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:52.392859 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:52.393159 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:52.393162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:52.393395 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:52.393508 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:52.393514 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:52.394297 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:52.394302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:52.394884 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:52.394891 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:52.395385 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:52.395389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:52.395474 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:52.396035 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:52.396046 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:52.397874 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":123,"threads_started":1}
W20260928 06:45:52.398111 16560 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:52.392500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:34861 (local address 127.7.48.62:41517)
0928 06:45:52.392671 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:52.392676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:52.392840 (+ 164us) server_negotiation.cc:407] Connection header received
0928 06:45:52.392975 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:52.392979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:52.393030 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:52.393125 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:52.393643 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:52.394180 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:52.395027 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:52.395271 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:52.395382 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:52.398007 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.2:34861: BlockingRecv error: recv got EOF from 127.7.48.2:34861 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 06:45:52 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:52 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:52.602279 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:52.596985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.3:58145)
0928 06:45:52.597275 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:52.597289 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:52.597379 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:52.597602 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:52.597604 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:52.597804 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:52.597899 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:52.597904 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:52.598625 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:52.598627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:52.599162 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:52.599168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:52.599584 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:52.599586 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:52.599654 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:52.600166 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:52.600178 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:52.602129 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":106,"threads_started":1}
W20260928 06:45:52.602418 16560 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:52.597218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:58145 (local address 127.7.48.62:41517)
0928 06:45:52.597367 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:52.597371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:52.597395 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:45:52.597439 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:52.597441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:52.597485 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:52.597559 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:52.598014 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:52.598528 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:52.599264 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:52.599475 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:52.599559 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:52.602292 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.3:58145: BlockingRecv error: recv got EOF from 127.7.48.3:58145 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 06:45:53 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:53 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:53.205231 16563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:53.199586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.1:59499)
0928 06:45:53.200005 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:53.200020 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:53.200131 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:53.200398 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:53.200401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:53.200602 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:53.200701 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:53.200707 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:53.201433 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:53.201435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:53.201977 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:53.201984 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:53.202401 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:53.202403 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:53.202484 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:53.203087 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:53.203100 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:53.205088 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":122,"threads_started":1}
W20260928 06:45:53.205391 16564 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:53.199727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:59499 (local address 127.7.48.62:41517)
0928 06:45:53.200023 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:53.200027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:53.200097 (+ 70us) server_negotiation.cc:407] Connection header received
0928 06:45:53.200223 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:53.200225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:53.200281 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:53.200361 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:53.200807 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:53.201337 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:53.202077 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:53.202300 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:53.202374 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:53.205253 (+ 2879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.1:59499: BlockingRecv error: recv got EOF from 127.7.48.1:59499 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":63,"threads_started":1}
Sep 28 06:45:53 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:53 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:53.404898 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:53.399010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.2:39931)
0928 06:45:53.399329 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:53.399351 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:53.399500 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:53.399786 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:53.399789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:53.400042 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:53.400138 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:53.400144 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:53.400927 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:53.400932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:53.401509 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:53.401516 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:53.401992 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:53.401995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:53.402091 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:53.402697 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:53.402708 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:53.404713 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":75,"threads_started":1}
W20260928 06:45:53.404999 16564 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:53.399090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:39931 (local address 127.7.48.62:41517)
0928 06:45:53.399260 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:53.399265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:53.399441 (+ 176us) server_negotiation.cc:407] Connection header received
0928 06:45:53.399622 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:53.399624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:53.399671 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:53.399764 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:53.400263 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:53.400809 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:53.401660 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:53.401876 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:53.401989 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:53.404857 (+ 2868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.2:39931: BlockingRecv error: recv got EOF from 127.7.48.2:39931 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 06:45:53 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:53 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:45:53.609273 16564 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:53.603519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:41915 (local address 127.7.48.62:41517)
0928 06:45:53.603705 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:53.603710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:53.603841 (+ 131us) server_negotiation.cc:407] Connection header received
0928 06:45:53.604088 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:53.604092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:53.604143 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:53.604250 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:53.604765 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:53.605380 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:53.606200 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:53.606434 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:53.606546 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:53.609173 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.3:41915: BlockingRecv error: recv got EOF from 127.7.48.3:41915 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 06:45:53.609265 16566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:53.603440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.3:41915)
0928 06:45:53.603732 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:53.603754 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:53.603939 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:53.604272 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:53.604275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:53.604507 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:53.604632 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:53.604639 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:53.605494 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:53.605499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:53.606068 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:53.606075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:53.606549 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:53.606552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:53.606662 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:53.607171 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:53.607183 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:53.609043 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":81,"threads_started":1}
Sep 28 06:45:54 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:54 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:54.211525 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:54.206051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.1:48539)
0928 06:45:54.206342 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:54.206356 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:54.206446 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:54.206710 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:54.206712 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:54.206893 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:54.206991 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:54.206996 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:54.207719 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:54.207722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:54.208305 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:54.208313 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:54.208720 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:54.208722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:54.208805 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:54.209349 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:54.209365 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:54.211355 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":100,"threads_started":1}
W20260928 06:45:54.211637 16568 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:54.206106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:48539 (local address 127.7.48.62:41517)
0928 06:45:54.206404 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:54.206408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:54.206422 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:45:54.206542 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:54.206545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:54.206588 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:54.206668 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:54.207094 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:54.207620 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:54.208399 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:54.208617 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:54.208693 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:54.211501 (+ 2808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.1:48539: BlockingRecv error: recv got EOF from 127.7.48.1:48539 (error 108)
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":139,"threads_started":1}
Sep 28 06:45:54 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:54 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:54.411398 16569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:54.405717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.2:41577)
0928 06:45:54.406009 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:54.406031 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:54.406175 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:54.406444 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:54.406447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:54.406688 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:54.406779 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:54.406785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:54.407529 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:54.407534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:54.408292 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:54.408302 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:54.408772 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:54.408776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:54.408863 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:54.409379 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:54.409390 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:54.411280 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":85,"threads_started":1}
W20260928 06:45:54.411590 16568 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:54.405791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:41577 (local address 127.7.48.62:41517)
0928 06:45:54.405968 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:54.405973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:54.406113 (+ 140us) server_negotiation.cc:407] Connection header received
0928 06:45:54.406279 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:54.406285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:54.406333 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:54.406425 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:54.406902 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:54.407417 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:54.408416 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:54.408656 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:54.408770 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:54.411428 (+ 2658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.2:41577: BlockingRecv error: recv got EOF from 127.7.48.2:41577 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 06:45:54 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:54 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:54.616530 16570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:54.610122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.3:53945)
0928 06:45:54.610488 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:54.610505 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:54.610663 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:54.610955 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:54.610958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:54.611189 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:54.611284 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:54.611289 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:54.612203 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:54.612365 (+ 162us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:54.613072 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:54.613083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:54.613665 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:54.613669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:54.613759 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:54.614338 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:54.614353 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:54.616413 (+ 2060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":105,"threads_started":1}
W20260928 06:45:54.616667 16568 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:54.610330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:53945 (local address 127.7.48.62:41517)
0928 06:45:54.610488 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:54.610493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:54.610599 (+ 106us) server_negotiation.cc:407] Connection header received
0928 06:45:54.610791 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:54.610793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:54.610838 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:54.610927 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:54.611409 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:54.612073 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:54.613210 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:54.613544 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:54.613667 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:54.616544 (+ 2877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.3:53945: BlockingRecv error: recv got EOF from 127.7.48.3:53945 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 06:45:55 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:55 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:55.218132 16571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:55.212375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.1:38001)
0928 06:45:55.212646 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:55.212664 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:55.212789 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:55.213042 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:55.213045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:55.213227 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:55.213324 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:55.213330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:55.214086 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:55.214089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:55.214815 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:55.214826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:55.215278 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:55.215280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:55.215394 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:55.215992 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:55.216006 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:55.218014 (+ 2008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":100,"threads_started":1}
W20260928 06:45:55.218350 16572 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:55.212464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:38001 (local address 127.7.48.62:41517)
0928 06:45:55.212721 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:55.212725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:55.212741 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:45:55.212877 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:55.212880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:55.212924 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:55.213003 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:55.213440 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:55.213982 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:55.214930 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:55.215179 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:55.215256 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:55.218195 (+ 2939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.1:38001: BlockingRecv error: recv got EOF from 127.7.48.1:38001 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"spinlock_wait_cycles":896,"thread_start_us":165,"threads_started":1}
Sep 28 06:45:55 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:55 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:55.418267 16573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:55.412368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.2:44755)
0928 06:45:55.412670 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:55.412688 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:55.412841 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:55.413155 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:55.413158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:55.413352 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:55.413466 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:55.413471 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:55.414219 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:55.414222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:55.414891 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:55.414906 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:55.415387 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:55.415390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:55.415454 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:55.416061 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:55.416073 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:55.418064 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":126,"threads_started":1}
W20260928 06:45:55.418301 16572 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:55.412533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:44755 (local address 127.7.48.62:41517)
0928 06:45:55.412685 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:55.412689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:55.412761 (+ 72us) server_negotiation.cc:407] Connection header received
0928 06:45:55.412941 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:55.412944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:55.413004 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:55.413078 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:55.413585 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:55.414120 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:55.415007 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:55.415234 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:55.415383 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:55.418193 (+ 2810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.2:44755: BlockingRecv error: recv got EOF from 127.7.48.2:44755 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 06:45:55 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:55 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:55.626417 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:55.617571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.3:39087)
0928 06:45:55.617976 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:55.617993 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:55.618118 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:55.618448 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:55.618452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:55.618794 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:55.618923 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:55.618931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:55.619681 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:55.619684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:55.620373 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:55.620382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:55.620808 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:55.620810 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:55.620870 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:55.624340 (+ 3470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:55.624353 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:55.626297 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":120,"threads_started":1}
W20260928 06:45:55.626627 16572 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:55.617679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:39087 (local address 127.7.48.62:41517)
0928 06:45:55.617845 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:55.617850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:55.618080 (+ 230us) server_negotiation.cc:407] Connection header received
0928 06:45:55.618234 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:55.618237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:55.618300 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:55.618386 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:55.619050 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:55.619587 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:55.620483 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:55.620708 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:55.620787 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:55.626494 (+ 5707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.3:39087: BlockingRecv error: recv got EOF from 127.7.48.3:39087 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 06:45:56 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:56 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:56.224691 16575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:56.218944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.1:42251)
0928 06:45:56.219233 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:56.219247 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:56.219434 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:56.219746 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:56.219749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:56.219984 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:56.220085 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:56.220091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:56.220875 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:56.220880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:56.221422 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:56.221430 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:56.221905 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:56.221908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:56.221997 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:56.222615 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:56.222627 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:56.224565 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":103,"threads_started":1}
W20260928 06:45:56.224812 16576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:56.218998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:42251 (local address 127.7.48.62:41517)
0928 06:45:56.219250 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:56.219254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:56.219393 (+ 139us) server_negotiation.cc:407] Connection header received
0928 06:45:56.219552 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:56.219557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:56.219617 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:56.219713 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:56.220212 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:56.220764 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:56.221544 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:56.221782 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:56.221887 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:56.224707 (+ 2820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.1:42251: BlockingRecv error: recv got EOF from 127.7.48.1:42251 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":113,"threads_started":1}
Sep 28 06:45:56 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:56 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:56.424695 16577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:56.419185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.2:46463)
0928 06:45:56.419468 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:56.419481 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:56.419659 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:56.419978 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:56.419981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:56.420204 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:56.420299 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:56.420305 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:56.421081 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:56.421086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:56.421655 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:56.421663 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:56.422125 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:56.422129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:56.422222 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:56.422715 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:56.422726 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:56.424575 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":85,"threads_started":1}
W20260928 06:45:56.424890 16576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:56.419266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:46463 (local address 127.7.48.62:41517)
0928 06:45:56.419418 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:56.419422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:56.419606 (+ 184us) server_negotiation.cc:407] Connection header received
0928 06:45:56.419767 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:56.419770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:56.419830 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:56.419958 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:56.420423 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:56.420965 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:56.421777 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:56.422013 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:56.422121 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:56.424728 (+ 2607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.2:46463: BlockingRecv error: recv got EOF from 127.7.48.2:46463 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 06:45:56 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:56 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:56.633137 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:56.627380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.3:49525)
0928 06:45:56.627680 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:56.627694 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:56.627803 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:56.628070 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:56.628072 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:56.628264 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:56.628359 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:56.628364 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:56.629442 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:56.629446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:56.630093 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:56.630102 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:56.630652 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:56.630655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:56.630717 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:56.631203 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:56.631214 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:56.633024 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":72,"threads_started":1}
W20260928 06:45:56.633278 16576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:56.627550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:49525 (local address 127.7.48.62:41517)
0928 06:45:56.627712 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:56.627716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:56.627771 (+ 55us) server_negotiation.cc:407] Connection header received
0928 06:45:56.627908 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:56.627910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:56.627959 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:56.628030 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:56.628506 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:56.629277 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:56.630216 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:56.630535 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:56.630646 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:56.633166 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.3:49525: BlockingRecv error: recv got EOF from 127.7.48.3:49525 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 28 06:45:57 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:57 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:57.231359 16579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:57.225703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.1:57571)
0928 06:45:57.226103 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:57.226120 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:57.226245 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:57.226578 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:57.226581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:57.226761 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:57.226861 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:57.226866 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:57.227559 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:57.227561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:57.228195 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:57.228204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:57.228629 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:57.228631 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:57.228692 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:57.229277 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:57.229289 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:57.231240 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":104,"threads_started":1}
W20260928 06:45:57.231519 16580 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:57.225777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:57571 (local address 127.7.48.62:41517)
0928 06:45:57.226056 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:57.226062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:57.226199 (+ 137us) server_negotiation.cc:407] Connection header received
0928 06:45:57.226361 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:57.226364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:57.226427 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:57.226539 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:57.226957 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:57.227468 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:57.228287 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:57.228529 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:57.228608 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:57.231388 (+ 2780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.1:57571: BlockingRecv error: recv got EOF from 127.7.48.1:57571 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
Sep 28 06:45:57 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:57 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:57.430713 16581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:57.425454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.2:54083)
0928 06:45:57.425819 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:57.425833 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:57.425934 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:57.426169 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:57.426171 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:57.426352 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:57.426451 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:57.426456 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:57.427145 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:57.427147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:57.427712 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:57.427719 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:57.428184 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:57.428187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:57.428251 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:57.428732 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:57.428743 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:57.430594 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":90,"threads_started":1}
W20260928 06:45:57.430837 16580 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:57.425499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:54083 (local address 127.7.48.62:41517)
0928 06:45:57.425641 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:57.425646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:57.425908 (+ 262us) server_negotiation.cc:407] Connection header received
0928 06:45:57.426014 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:57.426016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:57.426064 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:57.426139 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:57.426557 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:57.427054 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:57.427808 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:57.428086 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:57.428173 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:57.430717 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.2:54083: BlockingRecv error: recv got EOF from 127.7.48.2:54083 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 06:45:57 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:57 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:57.639338 16582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:57.633943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.3:37109)
0928 06:45:57.634311 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:57.634335 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:57.634466 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:57.634784 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:57.634786 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:57.634969 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:57.635064 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:57.635068 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:57.635783 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:57.635785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:57.636408 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:57.636416 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:57.636818 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:57.636820 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:57.636878 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:57.637375 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:57.637386 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:57.639222 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":112,"threads_started":1}
W20260928 06:45:57.639521 16580 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:57.634023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:37109 (local address 127.7.48.62:41517)
0928 06:45:57.634190 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:57.634195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:57.634415 (+ 220us) server_negotiation.cc:407] Connection header received
0928 06:45:57.634608 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:57.634611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:57.634670 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:57.634745 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:57.635166 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:57.635691 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:57.636511 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:57.636726 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:57.636799 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:57.639362 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.3:37109: BlockingRecv error: recv got EOF from 127.7.48.3:37109 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 06:45:58 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:58 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:58.237447 16583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:58.232221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.1:58143)
0928 06:45:58.232523 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:58.232536 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:58.232630 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:58.232885 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:58.232887 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:58.233067 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:58.233175 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:58.233181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:58.233971 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:58.233974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:58.234527 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:58.234535 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:58.234946 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:58.234948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:58.235007 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:58.235522 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:58.235533 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:58.237327 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":108,"threads_started":1}
W20260928 06:45:58.237751 16584 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:58.232378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:58143 (local address 127.7.48.62:41517)
0928 06:45:58.232607 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:58.232610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:58.232624 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:45:58.232724 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:58.232726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:58.232771 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:58.232847 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:58.233279 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:58.233871 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:58.234634 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:58.234851 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:58.234925 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:58.237473 (+ 2548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.1:58143: BlockingRecv error: recv got EOF from 127.7.48.1:58143 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":67,"threads_started":1}
Sep 28 06:45:58 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:58 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:58.437109 16585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:58.431604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.2:33533)
0928 06:45:58.431922 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:58.431940 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:58.432053 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:58.432294 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:58.432297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:58.432480 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:58.432578 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:58.432584 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:58.433420 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:58.433422 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:58.434030 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:58.434042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:58.434530 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:58.434532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:58.434594 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:58.435082 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:58.435093 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:58.436957 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":72,"threads_started":1}
W20260928 06:45:58.437247 16584 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:58.431741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:33533 (local address 127.7.48.62:41517)
0928 06:45:58.431934 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:58.431937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:58.432019 (+ 82us) server_negotiation.cc:407] Connection header received
0928 06:45:58.432138 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:58.432141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:58.432187 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:58.432256 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:58.432677 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:58.433326 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:58.434139 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:58.434428 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:58.434507 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:58.437103 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.2:33533: BlockingRecv error: recv got EOF from 127.7.48.2:33533 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 06:45:58 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:58 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:58.645576 16586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:58.640232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.3:51821)
0928 06:45:58.640510 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:58.640524 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:58.640625 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:58.640861 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:58.640863 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:58.641046 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:58.641150 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:58.641155 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:58.641888 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:58.641890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:58.642446 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:58.642452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:58.642855 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:58.642857 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:58.642917 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:58.643426 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:58.643438 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:58.645389 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":105,"threads_started":1}
W20260928 06:45:58.645797 16584 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:58.640317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:51821 (local address 127.7.48.62:41517)
0928 06:45:58.640466 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:58.640470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:58.640574 (+ 104us) server_negotiation.cc:407] Connection header received
0928 06:45:58.640704 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:58.640706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:58.640754 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:58.640820 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:58.641271 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:58.641794 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:58.642543 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:58.642759 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:58.642833 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:58.645558 (+ 2725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.3:51821: BlockingRecv error: recv got EOF from 127.7.48.3:51821 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 06:45:59 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:59 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:59.243952 16587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:59.238473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.1:36481)
0928 06:45:59.238766 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:59.238779 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:59.238871 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:59.239125 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:59.239127 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:59.239312 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:59.239406 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:59.239411 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:59.240341 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:59.240347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:59.241017 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:59.241027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:59.241448 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:59.241450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:59.241506 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:59.241974 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:59.241985 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:59.243728 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":111,"threads_started":1}
W20260928 06:45:59.243994 16588 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:59.238545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:36481 (local address 127.7.48.62:41517)
0928 06:45:59.238807 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:59.238811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:59.238826 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:45:59.238963 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:59.238965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:59.239009 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:59.239095 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:59.239532 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:59.240213 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:59.241137 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:59.241355 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:59.241434 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:59.243866 (+ 2432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.1:36481: BlockingRecv error: recv got EOF from 127.7.48.1:36481 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":82,"threads_started":1}
Sep 28 06:45:59 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:59 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:59.443799 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:59.438135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.2:45089)
0928 06:45:59.438454 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:59.438471 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:59.438610 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:59.438920 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:59.438923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:59.439178 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:59.439313 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:59.439321 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:59.440236 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:59.440239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:59.440784 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:59.440791 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:59.441189 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:59.441191 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:59.441252 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:59.441821 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:59.441838 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:59.443682 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":108,"threads_started":1}
W20260928 06:45:59.444080 16588 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:59.438225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:45089 (local address 127.7.48.62:41517)
0928 06:45:59.438427 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:59.438431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:59.438568 (+ 137us) server_negotiation.cc:407] Connection header received
0928 06:45:59.438708 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:59.438711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:59.438770 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:59.438860 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:59.439437 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:59.440141 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:59.440882 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:59.441090 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:59.441164 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:59.443815 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.2:45089: BlockingRecv error: recv got EOF from 127.7.48.2:45089 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 28 06:45:59 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:45:59 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:45:59.652447 16590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:45:59.646418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.3:33529)
0928 06:45:59.646808 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0928 06:45:59.646823 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:45:59.646970 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:45:59.647271 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:45:59.647274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:45:59.647497 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:45:59.647591 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:59.647597 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:59.648439 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:59.648444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:59.649299 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:45:59.649310 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:45:59.649894 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:45:59.649897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:45:59.649958 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:59.650468 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:45:59.650479 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:45:59.652327 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":157,"threads_started":1}
W20260928 06:45:59.652588 16588 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:45:59.646576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:33529 (local address 127.7.48.62:41517)
0928 06:45:59.646752 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 06:45:59.646757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:45:59.646916 (+ 159us) server_negotiation.cc:407] Connection header received
0928 06:45:59.647086 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:45:59.647089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:45:59.647152 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:45:59.647246 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:45:59.647724 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:59.648329 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:59.649447 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:45:59.649789 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:45:59.649873 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:45:59.652468 (+ 2595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.3:33529: BlockingRecv error: recv got EOF from 127.7.48.3:33529 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 06:46:00 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:46:00 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:46:00.249862 16591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:00.244730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.1:39323)
0928 06:46:00.245013 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:00.245027 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:00.245112 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:00.245325 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:00.245327 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:00.245503 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:00.245597 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:00.245602 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:00.246328 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:00.246330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:00.246888 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:00.246896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:00.247310 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:00.247313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:00.247373 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:00.247863 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:46:00.247902 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:00.249745 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
W20260928 06:46:00.250025 16592 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:46:00.244821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:39323 (local address 127.7.48.62:41517)
0928 06:46:00.245100 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:00.245103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:00.245116 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:46:00.245195 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:00.245197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:00.245239 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:00.245313 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:00.245708 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:00.246234 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:00.246990 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:00.247214 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:00.247288 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:00.249885 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.1:39323: BlockingRecv error: recv got EOF from 127.7.48.1:39323 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":83,"threads_started":1}
Sep 28 06:46:00 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:46:00 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:46:00.450280 16593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:00.444839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.2:47045)
0928 06:46:00.445131 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:00.445146 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:00.445243 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:00.445477 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:00.445479 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:00.445681 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:00.445780 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:00.445786 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:00.446538 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:00.446541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:00.447097 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:00.447105 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:00.447518 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:00.447521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:00.447596 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:00.448253 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:46:00.448275 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:00.450160 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":102,"threads_started":1}
W20260928 06:46:00.450431 16592 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:46:00.444884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:47045 (local address 127.7.48.62:41517)
0928 06:46:00.445052 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:00.445056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:00.445222 (+ 166us) server_negotiation.cc:407] Connection header received
0928 06:46:00.445313 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:00.445315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:00.445359 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:00.445429 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:00.445921 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:00.446434 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:00.447202 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:00.447418 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:00.447493 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:00.450309 (+ 2816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.2:47045: BlockingRecv error: recv got EOF from 127.7.48.2:47045 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 06:46:00 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:46:00 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:46:00.662307 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:00.653380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.3:59555)
0928 06:46:00.653664 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:00.653678 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:00.653760 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:00.654008 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:00.654010 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:00.654191 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:00.654283 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:00.654288 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:00.654980 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:00.654983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:00.655522 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:00.655528 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:00.656006 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:00.656008 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:00.656072 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:00.659878 (+ 3806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:46:00.659923 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:00.662137 (+ 2214us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":81,"threads_started":1}
W20260928 06:46:00.662367 16592 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:46:00.653541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:59555 (local address 127.7.48.62:41517)
0928 06:46:00.653707 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:00.653711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:00.653725 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:46:00.653849 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:00.653851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:00.653894 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:00.653968 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:00.654382 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:00.654890 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:00.655623 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:00.655836 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:00.655974 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:00.662265 (+ 6291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.3:59555: BlockingRecv error: recv got EOF from 127.7.48.3:59555 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 28 06:46:01 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:46:01 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:46:01.256487 16595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:01.250835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.1:49011)
0928 06:46:01.251130 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:01.251146 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:01.251266 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:01.251599 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:01.251602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:01.251914 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:01.252048 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:01.252055 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:01.252808 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:01.252811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:01.253369 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:01.253377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:01.253804 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:01.253806 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:01.253872 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:01.254385 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:46:01.254396 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:01.256346 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":112,"threads_started":1}
W20260928 06:46:01.256660 16596 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:46:01.250872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:49011 (local address 127.7.48.62:41517)
0928 06:46:01.251140 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:01.251143 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:01.251206 (+ 63us) server_negotiation.cc:407] Connection header received
0928 06:46:01.251372 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:01.251376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:01.251434 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:01.251543 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:01.252156 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:01.252696 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:01.253484 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:01.253703 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:01.253781 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:01.256517 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.1:49011: BlockingRecv error: recv got EOF from 127.7.48.1:49011 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":98,"threads_started":1}
Sep 28 06:46:01 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:46:01 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:46:01.456347 16597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:01.451122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.2:52915)
0928 06:46:01.451429 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:01.451443 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:01.451540 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:01.451775 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:01.451777 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:01.452022 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:01.452126 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:01.452132 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:01.452822 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:01.452824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:01.453374 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:01.453381 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:01.453781 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:01.453784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:01.453844 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:01.454405 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:46:01.454417 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:01.456228 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":136,"threads_started":1}
W20260928 06:46:01.456525 16596 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:46:01.451179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:52915 (local address 127.7.48.62:41517)
0928 06:46:01.451313 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:01.451317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:01.451513 (+ 196us) server_negotiation.cc:407] Connection header received
0928 06:46:01.451625 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:01.451627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:01.451670 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:01.451736 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:01.452226 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:01.452732 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:01.453478 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:01.453688 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:01.453760 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:01.456364 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.2:52915: BlockingRecv error: recv got EOF from 127.7.48.2:52915 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 06:46:01 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:46:01 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:46:01.668722 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:01.663388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.3:43489)
0928 06:46:01.663711 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:01.663724 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:01.663825 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:01.664071 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:01.664073 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:01.664257 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:01.664352 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:01.664358 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:01.665060 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:01.665063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:01.665619 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:01.665625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:01.666025 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:01.666027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:01.666089 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:01.666654 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:46:01.666666 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:01.668576 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":127,"threads_started":1}
W20260928 06:46:01.668814 16596 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:46:01.663461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:43489 (local address 127.7.48.62:41517)
0928 06:46:01.663636 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:01.663642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:01.663798 (+ 156us) server_negotiation.cc:407] Connection header received
0928 06:46:01.663916 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:01.663918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:01.663965 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:01.664054 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:01.664454 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:01.664968 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:01.665720 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:01.665930 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:01.666004 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:01.668708 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.3:43489: BlockingRecv error: recv got EOF from 127.7.48.3:43489 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 28 06:46:02 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:46:02 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:46:02.263214 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:02.257387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.1:40507)
0928 06:46:02.257667 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:02.257684 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:02.257835 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:02.258119 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:02.258122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:02.258382 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:02.258563 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:02.258570 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:02.259403 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:02.259408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:02.260074 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:02.260082 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:02.260560 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:02.260563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:02.260649 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:02.261166 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:46:02.261179 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:02.263036 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
W20260928 06:46:02.263300 16600 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:46:02.257509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:40507 (local address 127.7.48.62:41517)
0928 06:46:02.257757 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:02.257760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:02.257774 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:46:02.257923 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:02.257925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:02.257975 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:02.258092 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:02.258714 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:02.259285 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:02.260188 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:02.260440 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:02.260552 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:02.263171 (+ 2619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.1:40507: BlockingRecv error: recv got EOF from 127.7.48.1:40507 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":97,"threads_started":1}
Sep 28 06:46:02 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:46:02 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:46:02.462306 16601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:02.457203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.2:56161)
0928 06:46:02.457571 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:02.457584 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:02.457682 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:02.457873 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:02.457875 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:02.458051 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:02.458155 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:02.458159 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:02.458833 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:02.458835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:02.459390 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:02.459398 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:02.459787 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:02.459790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:02.459852 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:02.460357 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:46:02.460368 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:02.462130 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":111,"threads_started":1}
W20260928 06:46:02.462361 16600 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:46:02.457238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:56161 (local address 127.7.48.62:41517)
0928 06:46:02.457378 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:02.457383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:02.457658 (+ 275us) server_negotiation.cc:407] Connection header received
0928 06:46:02.457748 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:02.457750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:02.457798 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:02.457862 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:02.458248 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:02.458744 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:02.459490 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:02.459696 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:02.459767 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:02.462255 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.2:56161: BlockingRecv error: recv got EOF from 127.7.48.2:56161 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 06:46:02 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:46:02 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:46:02.674906 16602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:02.669598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.3:48799)
0928 06:46:02.669890 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:02.669904 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:02.670019 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:02.670266 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:02.670268 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:02.670455 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:02.670550 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:02.670555 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:02.671252 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:02.671254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:02.671791 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:02.671798 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:02.672278 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:02.672281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:02.672342 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:02.672817 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:46:02.672828 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:02.674714 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":89,"threads_started":1}
W20260928 06:46:02.674939 16600 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:46:02.669740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:48799 (local address 127.7.48.62:41517)
0928 06:46:02.669920 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:02.669923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:02.669990 (+ 67us) server_negotiation.cc:407] Connection header received
0928 06:46:02.670113 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:02.670115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:02.670162 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:02.670228 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:02.670652 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:02.671160 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:02.671911 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:02.672182 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:02.672257 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:02.674838 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.3:48799: BlockingRecv error: recv got EOF from 127.7.48.3:48799 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 28 06:46:03 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:46:03 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:46:03.269498 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:03.264086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.1:40859)
0928 06:46:03.264360 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:03.264375 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:03.264477 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:03.264719 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:03.264721 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:03.264901 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:03.264998 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:03.265003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:03.265733 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:03.265735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:03.266299 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:03.266306 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:03.266706 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:03.266708 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:03.266767 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:03.267316 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:46:03.267328 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:03.269307 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":95,"threads_started":1}
W20260928 06:46:03.269563 16604 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:46:03.264141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:40859 (local address 127.7.48.62:41517)
0928 06:46:03.264384 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:03.264387 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:03.264449 (+ 62us) server_negotiation.cc:407] Connection header received
0928 06:46:03.264561 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:03.264564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:03.264609 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:03.264687 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:03.265104 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:03.265642 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:03.266397 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:03.266614 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:03.266688 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:03.269442 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.1:40859: BlockingRecv error: recv got EOF from 127.7.48.1:40859 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":64,"threads_started":1}
Sep 28 06:46:03 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:46:03 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:46:03.468659 16605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:03.463073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.2:57379)
0928 06:46:03.463398 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:03.463416 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:03.463570 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:03.463901 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:03.463904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:03.464141 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:03.464237 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:03.464243 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:03.464996 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:03.465001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:03.465564 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:03.465571 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:03.466037 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:03.466041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:03.466129 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:03.466668 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:46:03.466680 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:03.468543 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":105,"threads_started":1}
W20260928 06:46:03.468796 16604 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:46:03.463228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:57379 (local address 127.7.48.62:41517)
0928 06:46:03.463399 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:03.463405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:03.463504 (+ 99us) server_negotiation.cc:407] Connection header received
0928 06:46:03.463692 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:03.463695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:03.463747 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:03.463848 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:03.464359 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:03.464886 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:03.465693 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:03.465923 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:03.466032 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:03.468681 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.2:57379: BlockingRecv error: recv got EOF from 127.7.48.2:57379 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 06:46:03 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:46:03 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:46:03.681586 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:03.675753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.3:52379)
0928 06:46:03.676074 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:03.676088 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:03.676222 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:03.676502 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:03.676505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:03.676730 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:03.676821 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:03.676828 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:03.677645 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:03.677649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:03.678215 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:03.678222 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:03.678691 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:03.678694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:03.678781 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:03.679286 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:46:03.679296 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:03.681413 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":146,"threads_started":1}
W20260928 06:46:03.681645 16604 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:46:03.675817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:52379 (local address 127.7.48.62:41517)
0928 06:46:03.676015 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:03.676020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:03.676172 (+ 152us) server_negotiation.cc:407] Connection header received
0928 06:46:03.676331 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:03.676334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:03.676389 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:03.676477 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:03.676942 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:03.677513 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:03.678340 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:03.678574 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:03.678684 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:03.681533 (+ 2849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.3:52379: BlockingRecv error: recv got EOF from 127.7.48.3:52379 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 28 06:46:04 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:46:04 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:46:04.275755 16607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:04.270377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.1:39973)
0928 06:46:04.270692 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:04.270706 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:04.270799 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:04.271058 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:04.271060 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:04.271246 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:04.271342 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:04.271348 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:04.272172 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:04.272174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:04.272733 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:04.272742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:04.273175 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:04.273177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:04.273249 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:04.273809 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:46:04.273821 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:04.275615 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":113,"threads_started":1}
W20260928 06:46:04.275944 16608 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:46:04.270458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:39973 (local address 127.7.48.62:41517)
0928 06:46:04.270775 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:04.270778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:04.270791 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:46:04.270893 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:04.270895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:04.270940 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:04.271020 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:04.271466 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:04.272073 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:04.272846 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:04.273072 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:04.273151 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:04.275781 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.1:39973: BlockingRecv error: recv got EOF from 127.7.48.1:39973 (error 108)
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":84,"threads_started":1}
Sep 28 06:46:04 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:46:04 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:46:04.475301 16609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:04.469474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.2:47347)
0928 06:46:04.469782 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:04.469796 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:04.469941 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:04.470237 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:04.470240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:04.470455 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:04.470552 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:04.470557 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:04.471349 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:04.471353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:04.472003 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:04.472013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:04.472476 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:04.472480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:04.472563 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:04.473097 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:46:04.473108 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:04.475097 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":89,"threads_started":1}
W20260928 06:46:04.475399 16608 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:46:04.469547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:47347 (local address 127.7.48.62:41517)
0928 06:46:04.469717 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:04.469723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:04.469878 (+ 155us) server_negotiation.cc:407] Connection header received
0928 06:46:04.470055 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:04.470060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:04.470116 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:04.470211 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:04.470676 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:04.471234 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:04.472123 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:04.472352 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:04.472459 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:04.475267 (+ 2808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.2:47347: BlockingRecv error: recv got EOF from 127.7.48.2:47347 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 06:46:04 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:46:04 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.3@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:46:04.688849 16610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:04.682604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.3:43167)
0928 06:46:04.682942 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:04.682958 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:04.683109 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:04.683424 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:04.683427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:04.683653 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:04.683748 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:04.683754 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:04.684578 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:04.684583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:04.685148 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:04.685156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:04.685673 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:04.685676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:04.685767 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:04.686475 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:46:04.686491 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:04.688649 (+ 2158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":119,"threads_started":1}
W20260928 06:46:04.688886 16608 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:46:04.682688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:43167 (local address 127.7.48.62:41517)
0928 06:46:04.682869 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:04.682875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:04.683050 (+ 175us) server_negotiation.cc:407] Connection header received
0928 06:46:04.683249 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:04.683251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:04.683301 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:04.683393 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:04.683910 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:04.684455 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:04.685288 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:04.685550 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:04.685666 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:04.688777 (+ 3111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.3:43167: BlockingRecv error: recv got EOF from 127.7.48.3:43167 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 06:46:05 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:46:05 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.1@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:46:05.282574 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:05.276932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.1:53971)
0928 06:46:05.277239 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:05.277255 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:05.277345 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:05.277594 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:05.277596 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:05.277773 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:05.277868 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:05.277873 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:05.278598 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:05.278600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:05.279150 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:05.279157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:05.279567 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:05.279569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:05.279632 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:05.280201 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:46:05.280213 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:05.282415 (+ 2202us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":70,"threads_started":1}
W20260928 06:46:05.282723 16612 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:46:05.277019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:53971 (local address 127.7.48.62:41517)
0928 06:46:05.277302 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:05.277306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:05.277320 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:46:05.277433 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:05.277435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:05.277479 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:05.277557 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:05.277967 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:05.278501 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:05.279249 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:05.279467 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:05.279541 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:05.282564 (+ 3023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.1:53971: BlockingRecv error: recv got EOF from 127.7.48.1:53971 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":104,"threads_started":1}
Sep 28 06:46:05 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:46:05 dist-test-slave-twl6 krb5kdc[15554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.48.2@KRBTEST.COM for oryx/127.7.48.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:46:05.481794 16613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:05.476427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41517 (local address 127.7.48.2:45733)
0928 06:46:05.476702 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:05.476716 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:05.476807 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:05.477024 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:05.477026 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:05.477210 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:05.477307 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:05.477313 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:05.478020 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:05.478022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:05.478582 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:05.478590 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:05.479009 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:05.479011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:05.479070 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:05.479590 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:46:05.479601 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:05.481675 (+ 2074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.48.62:41517: Server oryx/127.7.48.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":107,"threads_started":1}
W20260928 06:46:05.481979 16612 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:46:05.476481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:45733 (local address 127.7.48.62:41517)
0928 06:46:05.476642 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:05.476648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:05.476769 (+ 121us) server_negotiation.cc:407] Connection header received
0928 06:46:05.476875 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:05.476877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:05.476921 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:05.476987 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:05.477409 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:05.477927 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:05.478686 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:05.478914 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:05.478988 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:05.481815 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.48.2:45733: BlockingRecv error: recv got EOF from 127.7.48.2:45733 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:46:05.578616 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 16137
I20260928 06:46:05.584755 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 16289
I20260928 06:46:05.590472 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 16421
I20260928 06:46:05.595762 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 16067
2026-09-28T06:46:05Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18012 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T06:46:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:46:05Z Disabled control of system clock
I20260928 06:46:05.626130 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:40725
--webserver_interface=127.7.48.62
--webserver_port=0
--builtin_ntp_servers=127.7.48.20:35431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:40725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 06:46:05.737088 16617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:46:05.737375 16617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:46:05.737413 16617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:46:05.741567 16617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:46:05.741631 16617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:46:05.741647 16617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:46:05.741664 16617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:46:05.741680 16617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:46:05.746464 16617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:35431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:40725
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:40725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:46:05.747547 16617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:46:05.748941 16617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:46:05.757998 16622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:05.758047 16625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:46:05.758025 16617 server_base.cc:1034] running on GCE node
W20260928 06:46:05.758002 16623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:46:05.758879 16617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:46:05.760097 16617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:46:05.761279 16617 hybrid_clock.cc:648] HybridClock initialized: now 1790577965761264 us; error 30 us; skew 500 ppm
I20260928 06:46:05.768220 16617 webserver.cc:460] Webserver started at http://127.7.48.62:39193/ using document root <none> and password file <none>
I20260928 06:46:05.768846 16617 fs_manager.cc:359] Metadata directory not provided
I20260928 06:46:05.768942 16617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:46:05.769189 16617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:46:05.771304 16617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "c1d87a246be84f9b9bec98f5a1d77a10"
format_stamp: "Formatted at 2026-09-28 06:46:05 on dist-test-slave-twl6"
server_key: "3863704721ee76b8976f9269bc5c3249"
server_key_iv: "a7bae58b2c19b2f988c8e6e0fb7d3503"
server_key_version: "encryptionkey@0"
I20260928 06:46:05.771936 16617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "c1d87a246be84f9b9bec98f5a1d77a10"
format_stamp: "Formatted at 2026-09-28 06:46:05 on dist-test-slave-twl6"
server_key: "3863704721ee76b8976f9269bc5c3249"
server_key_iv: "a7bae58b2c19b2f988c8e6e0fb7d3503"
server_key_version: "encryptionkey@0"
I20260928 06:46:05.775813 16617 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 06:46:05.778331 16631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:05.779556 16617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 06:46:05.779695 16617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "c1d87a246be84f9b9bec98f5a1d77a10"
format_stamp: "Formatted at 2026-09-28 06:46:05 on dist-test-slave-twl6"
server_key: "3863704721ee76b8976f9269bc5c3249"
server_key_iv: "a7bae58b2c19b2f988c8e6e0fb7d3503"
server_key_version: "encryptionkey@0"
I20260928 06:46:05.779803 16617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:46:05.807279 16617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:46:05.808126 16617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:46:05.808346 16617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:46:05.815446 16617 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:40725
I20260928 06:46:05.815454 16683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:40725 every 8 connection(s)
I20260928 06:46:05.816501 16617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:46:05.819375 16684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:46:05.822177 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 16617
I20260928 06:46:05.822314 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:46:05.822592 7360 external_mini_cluster.cc:1428] Setting key 12495a6d0bc45c92bd45b84396761863
I20260928 06:46:05.825670 16684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1d87a246be84f9b9bec98f5a1d77a10: Bootstrap starting.
I20260928 06:46:05.828374 16684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1d87a246be84f9b9bec98f5a1d77a10: Neither blocks nor log segments found. Creating new log.
I20260928 06:46:05.829371 16684 log.cc:824] T 00000000000000000000000000000000 P c1d87a246be84f9b9bec98f5a1d77a10: Log is configured to *not* fsync() on all Append() calls
I20260928 06:46:05.831728 16684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1d87a246be84f9b9bec98f5a1d77a10: No bootstrap required, opened a new log
I20260928 06:46:05.833396 16687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:05.824089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54676 (local address 127.7.48.62:40725)
0928 06:46:05.824798 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:05.824820 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:05.824862 (+ 42us) server_negotiation.cc:407] Connection header received
0928 06:46:05.825681 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:05.825707 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:05.826210 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:05.826569 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:05.827051 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:05.828480 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:05.829719 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:05.830353 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:05.830505 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:05.831321 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:46:05.831328 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:46:05.831344 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:46:05.831847 (+ 503us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:46:05.831947 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:46:05.831975 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:46:05.832088 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:05.832193 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 06:46:05.832477 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":119,"threads_started":1}
I20260928 06:46:05.836138 16684 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1d87a246be84f9b9bec98f5a1d77a10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d87a246be84f9b9bec98f5a1d77a10" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 40725 } }
I20260928 06:46:05.836351 16684 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1d87a246be84f9b9bec98f5a1d77a10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:46:05.836386 16684 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1d87a246be84f9b9bec98f5a1d77a10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d87a246be84f9b9bec98f5a1d77a10, State: Initialized, Role: FOLLOWER
I20260928 06:46:05.836752 16684 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1d87a246be84f9b9bec98f5a1d77a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d87a246be84f9b9bec98f5a1d77a10" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 40725 } }
I20260928 06:46:05.836860 16684 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1d87a246be84f9b9bec98f5a1d77a10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:46:05.836903 16684 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1d87a246be84f9b9bec98f5a1d77a10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:46:05.836973 16684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1d87a246be84f9b9bec98f5a1d77a10 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:46:05.837817 16684 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1d87a246be84f9b9bec98f5a1d77a10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d87a246be84f9b9bec98f5a1d77a10" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 40725 } }
I20260928 06:46:05.838104 16684 leader_election.cc:302] T 00000000000000000000000000000000 P c1d87a246be84f9b9bec98f5a1d77a10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1d87a246be84f9b9bec98f5a1d77a10; no voters:
I20260928 06:46:05.838341 16684 leader_election.cc:288] T 00000000000000000000000000000000 P c1d87a246be84f9b9bec98f5a1d77a10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:46:05.838490 16689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1d87a246be84f9b9bec98f5a1d77a10 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:46:05.838800 16689 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1d87a246be84f9b9bec98f5a1d77a10 [term 1 LEADER]: Becoming Leader. State: Replica: c1d87a246be84f9b9bec98f5a1d77a10, State: Running, Role: LEADER
I20260928 06:46:05.839133 16689 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1d87a246be84f9b9bec98f5a1d77a10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d87a246be84f9b9bec98f5a1d77a10" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 40725 } }
I20260928 06:46:05.839555 16684 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1d87a246be84f9b9bec98f5a1d77a10 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:46:05.840803 16691 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1d87a246be84f9b9bec98f5a1d77a10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1d87a246be84f9b9bec98f5a1d77a10. Latest consensus state: current_term: 1 leader_uuid: "c1d87a246be84f9b9bec98f5a1d77a10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d87a246be84f9b9bec98f5a1d77a10" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 40725 } } }
I20260928 06:46:05.840929 16691 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1d87a246be84f9b9bec98f5a1d77a10 [sys.catalog]: This master's current role is: LEADER
I20260928 06:46:05.841243 16690 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1d87a246be84f9b9bec98f5a1d77a10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1d87a246be84f9b9bec98f5a1d77a10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d87a246be84f9b9bec98f5a1d77a10" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 40725 } } }
I20260928 06:46:05.841404 16690 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1d87a246be84f9b9bec98f5a1d77a10 [sys.catalog]: This master's current role is: LEADER
I20260928 06:46:05.841403 16698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:46:05.844449 16698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:46:05.849968 16698 catalog_manager.cc:1347] Generated new cluster ID: cd1c9541ce9b4aa28778529502c770a6
I20260928 06:46:05.850060 16698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:46:05.858374 16698 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:46:05.859258 16698 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:46:05.865295 16698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1d87a246be84f9b9bec98f5a1d77a10: Generated new TSK 0
I20260928 06:46:05.866140 16698 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:46:05.879206 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:40725
--builtin_ntp_servers=127.7.48.20:35431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 06:46:05.993801 16708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:46:05.994084 16708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:46:05.994122 16708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:46:05.998257 16708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:46:05.998313 16708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:46:05.998384 16708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:46:06.003130 16708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:35431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:40725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:46:06.004304 16708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:46:06.005774 16708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:46:06.015606 16713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:06.015797 16714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:46:06.015867 16708 server_base.cc:1034] running on GCE node
W20260928 06:46:06.015628 16716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:46:06.016512 16708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:46:06.017201 16708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:46:06.018404 16708 hybrid_clock.cc:648] HybridClock initialized: now 1790577966018394 us; error 31 us; skew 500 ppm
I20260928 06:46:06.020517 16708 webserver.cc:460] Webserver started at http://127.7.48.1:35423/ using document root <none> and password file <none>
I20260928 06:46:06.021171 16708 fs_manager.cc:359] Metadata directory not provided
I20260928 06:46:06.021286 16708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:46:06.021544 16708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:46:06.023789 16708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "52b48f6ba31143a9bb5b3f01b25a39f6"
format_stamp: "Formatted at 2026-09-28 06:46:06 on dist-test-slave-twl6"
server_key: "f80b3da47ace7fd564847851fd94d843"
server_key_iv: "e02924e33cad3b269d11475f71d73715"
server_key_version: "encryptionkey@0"
I20260928 06:46:06.024477 16708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "52b48f6ba31143a9bb5b3f01b25a39f6"
format_stamp: "Formatted at 2026-09-28 06:46:06 on dist-test-slave-twl6"
server_key: "f80b3da47ace7fd564847851fd94d843"
server_key_iv: "e02924e33cad3b269d11475f71d73715"
server_key_version: "encryptionkey@0"
I20260928 06:46:06.028579 16708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 06:46:06.031275 16722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:06.032635 16708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:46:06.032787 16708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "52b48f6ba31143a9bb5b3f01b25a39f6"
format_stamp: "Formatted at 2026-09-28 06:46:06 on dist-test-slave-twl6"
server_key: "f80b3da47ace7fd564847851fd94d843"
server_key_iv: "e02924e33cad3b269d11475f71d73715"
server_key_version: "encryptionkey@0"
I20260928 06:46:06.032912 16708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:46:06.065552 16708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:46:06.066393 16708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:46:06.066641 16708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:46:06.067325 16708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:46:06.068463 16708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:46:06.068544 16708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:06.068629 16708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:46:06.068682 16708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:06.078459 16708 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:43263
I20260928 06:46:06.078475 16835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:43263 every 8 connection(s)
I20260928 06:46:06.079484 16708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:46:06.086318 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 16708
I20260928 06:46:06.086410 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
I20260928 06:46:06.086688 7360 external_mini_cluster.cc:1428] Setting key d221178e50e455ff4eae527bd7bef269
I20260928 06:46:06.088630 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:40725
--builtin_ntp_servers=127.7.48.20:35431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 06:46:06.089589 16687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:06.081476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:52957 (local address 127.7.48.62:40725)
0928 06:46:06.081624 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:06.081628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:06.082902 (+ 1274us) server_negotiation.cc:407] Connection header received
0928 06:46:06.084533 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:06.084536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:06.084600 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:06.084684 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:06.085562 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:06.086090 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:06.087501 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:06.087731 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:06.087845 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:06.089082 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:46:06.089086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:46:06.089091 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:46:06.089108 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:46:06.089128 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:46:06.089136 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:46:06.089213 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:06.089377 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 06:46:06.089442 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:46:06.090533 16838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:06.081907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:40725 (local address 127.7.48.1:52957)
0928 06:46:06.082734 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:06.082788 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:06.084306 (+ 1518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:06.084851 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:06.084858 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:06.084944 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:06.085437 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:06.085449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:06.086235 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:06.086243 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:06.087358 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:06.087372 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:06.087920 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:06.087925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:06.088106 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:06.088786 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:46:06.088788 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:06.088851 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:46:06.088927 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:46:06.089229 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:46:06.089238 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:46:06.089244 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:46:06.089377 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 06:46:06.089695 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":128,"threads_started":1}
I20260928 06:46:06.091796 16836 heartbeater.cc:347] Connected to a master server at 127.7.48.62:40725
I20260928 06:46:06.092072 16836 heartbeater.cc:464] Registering TS with master...
I20260928 06:46:06.092667 16836 heartbeater.cc:511] Master 127.7.48.62:40725 requested a full tablet report, sending...
I20260928 06:46:06.094209 16648 ts_manager.cc:194] Registered new tserver with Master: 52b48f6ba31143a9bb5b3f01b25a39f6 (127.7.48.1:43263)
I20260928 06:46:06.095358 16648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.48.1:52957
W20260928 06:46:06.203045 16839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:46:06.203317 16839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:46:06.203354 16839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:46:06.207468 16839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:46:06.207520 16839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:46:06.207592 16839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:46:06.212395 16839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:35431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:40725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:46:06.213531 16839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:46:06.214941 16839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:46:06.224411 16847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:06.224406 16844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:06.224406 16845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:46:06.224659 16839 server_base.cc:1034] running on GCE node
I20260928 06:46:06.225070 16839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:46:06.225692 16839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:46:06.226889 16839 hybrid_clock.cc:648] HybridClock initialized: now 1790577966226858 us; error 48 us; skew 500 ppm
I20260928 06:46:06.228713 16839 webserver.cc:460] Webserver started at http://127.7.48.2:42151/ using document root <none> and password file <none>
I20260928 06:46:06.229324 16839 fs_manager.cc:359] Metadata directory not provided
I20260928 06:46:06.229417 16839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:46:06.229652 16839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:46:06.231643 16839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "7e2f6456a09d47adb00d5bfa0fa04aad"
format_stamp: "Formatted at 2026-09-28 06:46:06 on dist-test-slave-twl6"
server_key: "992ac8006b21cdb0d59a864bf49a5044"
server_key_iv: "a1647b9e2b1a84ce8eaae9afe249f8dd"
server_key_version: "encryptionkey@0"
I20260928 06:46:06.232235 16839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "7e2f6456a09d47adb00d5bfa0fa04aad"
format_stamp: "Formatted at 2026-09-28 06:46:06 on dist-test-slave-twl6"
server_key: "992ac8006b21cdb0d59a864bf49a5044"
server_key_iv: "a1647b9e2b1a84ce8eaae9afe249f8dd"
server_key_version: "encryptionkey@0"
I20260928 06:46:06.235940 16839 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:46:06.238586 16853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:06.240012 16839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:46:06.240144 16839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "7e2f6456a09d47adb00d5bfa0fa04aad"
format_stamp: "Formatted at 2026-09-28 06:46:06 on dist-test-slave-twl6"
server_key: "992ac8006b21cdb0d59a864bf49a5044"
server_key_iv: "a1647b9e2b1a84ce8eaae9afe249f8dd"
server_key_version: "encryptionkey@0"
I20260928 06:46:06.240257 16839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:46:06.254544 16839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:46:06.255292 16839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:46:06.255491 16839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:46:06.256137 16839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:46:06.257084 16839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:46:06.257155 16839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:06.257229 16839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:46:06.257272 16839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:06.266631 16966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:37037 every 8 connection(s)
I20260928 06:46:06.266610 16839 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:37037
I20260928 06:46:06.267637 16839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:46:06.275763 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 16839
I20260928 06:46:06.275928 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:46:06.276221 7360 external_mini_cluster.cc:1428] Setting key b300e22a410be79affb0ac61deb07a6e
I20260928 06:46:06.276993 16687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:06.269615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:48545 (local address 127.7.48.62:40725)
0928 06:46:06.269772 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:06.269776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:06.270861 (+ 1085us) server_negotiation.cc:407] Connection header received
0928 06:46:06.272306 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:06.272308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:06.272356 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:06.272432 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:06.273326 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:06.273864 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:06.274949 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:06.275153 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:06.275227 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:06.276470 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:46:06.276479 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:46:06.276482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:46:06.276494 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:46:06.276521 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:46:06.276530 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:46:06.276609 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:06.276784 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 06:46:06.276848 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 06:46:06.277905 16969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:06.270035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:40725 (local address 127.7.48.2:48545)
0928 06:46:06.270705 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:06.270746 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:06.272104 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:06.272612 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:06.272619 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:06.272702 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:06.273202 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:06.273214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:06.273976 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:06.273978 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:06.274846 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:06.274854 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:06.275292 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:06.275294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:06.275445 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:06.276170 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:46:06.276172 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:06.276213 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:46:06.276275 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:46:06.276625 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:46:06.276634 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:46:06.276640 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:46:06.276770 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 06:46:06.277064 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":118,"threads_started":1}
I20260928 06:46:06.278281 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:40725
--builtin_ntp_servers=127.7.48.20:35431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 06:46:06.279081 16967 heartbeater.cc:347] Connected to a master server at 127.7.48.62:40725
I20260928 06:46:06.279307 16967 heartbeater.cc:464] Registering TS with master...
I20260928 06:46:06.279814 16967 heartbeater.cc:511] Master 127.7.48.62:40725 requested a full tablet report, sending...
I20260928 06:46:06.280933 16648 ts_manager.cc:194] Registered new tserver with Master: 7e2f6456a09d47adb00d5bfa0fa04aad (127.7.48.2:37037)
I20260928 06:46:06.281526 16648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.48.2:48545
W20260928 06:46:06.390965 16970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:46:06.391255 16970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:46:06.391330 16970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:46:06.395236 16970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:46:06.395318 16970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:46:06.395406 16970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:46:06.400005 16970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:35431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:40725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:46:06.401080 16970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:46:06.402541 16970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:46:06.412182 16970 server_base.cc:1034] running on GCE node
W20260928 06:46:06.412125 16976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:06.412132 16975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:06.412384 16978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:46:06.412812 16970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:46:06.413509 16970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:46:06.414695 16970 hybrid_clock.cc:648] HybridClock initialized: now 1790577966414660 us; error 48 us; skew 500 ppm
I20260928 06:46:06.416532 16970 webserver.cc:460] Webserver started at http://127.7.48.3:34703/ using document root <none> and password file <none>
I20260928 06:46:06.417150 16970 fs_manager.cc:359] Metadata directory not provided
I20260928 06:46:06.417243 16970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:46:06.417478 16970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:46:06.419508 16970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "be2c659f7158470bba0c9b60bb5162f1"
format_stamp: "Formatted at 2026-09-28 06:46:06 on dist-test-slave-twl6"
server_key: "61d505b2b58b7f81d582bc049f3b2d9e"
server_key_iv: "cb1a7886ebf7ef6f334e1773ba8b7037"
server_key_version: "encryptionkey@0"
I20260928 06:46:06.420151 16970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "be2c659f7158470bba0c9b60bb5162f1"
format_stamp: "Formatted at 2026-09-28 06:46:06 on dist-test-slave-twl6"
server_key: "61d505b2b58b7f81d582bc049f3b2d9e"
server_key_iv: "cb1a7886ebf7ef6f334e1773ba8b7037"
server_key_version: "encryptionkey@0"
I20260928 06:46:06.424016 16970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 06:46:06.426604 16984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:06.427857 16970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:46:06.428047 16970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "be2c659f7158470bba0c9b60bb5162f1"
format_stamp: "Formatted at 2026-09-28 06:46:06 on dist-test-slave-twl6"
server_key: "61d505b2b58b7f81d582bc049f3b2d9e"
server_key_iv: "cb1a7886ebf7ef6f334e1773ba8b7037"
server_key_version: "encryptionkey@0"
I20260928 06:46:06.428175 16970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:46:06.476155 16970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:46:06.476923 16970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:46:06.477144 16970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:46:06.477772 16970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:46:06.478766 16970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:46:06.478838 16970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:06.478914 16970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:46:06.478958 16970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:06.488391 16970 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:37831
I20260928 06:46:06.488426 17097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:37831 every 8 connection(s)
I20260928 06:46:06.489306 16970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:46:06.495411 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 16970
I20260928 06:46:06.495543 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:46:06.495795 7360 external_mini_cluster.cc:1428] Setting key 4bff2f989fa155abffa8962eb51107b4
I20260928 06:46:06.498854 16687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:06.491161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:57413 (local address 127.7.48.62:40725)
0928 06:46:06.491310 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:06.491314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:06.492428 (+ 1114us) server_negotiation.cc:407] Connection header received
0928 06:46:06.493819 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:06.493822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:06.493882 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:06.493987 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:06.494886 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:06.495445 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:06.496851 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:06.497117 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:06.497227 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:06.498451 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:46:06.498454 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:46:06.498456 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:46:06.498467 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:46:06.498488 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:46:06.498496 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:46:06.498561 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:06.498697 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 06:46:06.498748 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 06:46:06.499722 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:06.491597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:40725 (local address 127.7.48.3:57413)
0928 06:46:06.492272 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:06.492313 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:06.493603 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:06.494171 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:06.494177 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:06.494260 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:06.494749 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:06.494762 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:06.495568 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:06.495572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:06.496717 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:06.496728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:06.497291 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:06.497295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:06.497497 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:06.498219 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:46:06.498221 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:06.498261 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:46:06.498322 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:46:06.498575 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:46:06.498587 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:46:06.498591 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:46:06.498683 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 06:46:06.498893 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":112,"threads_started":1}
I20260928 06:46:06.500782 17098 heartbeater.cc:347] Connected to a master server at 127.7.48.62:40725
I20260928 06:46:06.500978 17098 heartbeater.cc:464] Registering TS with master...
I20260928 06:46:06.501418 17098 heartbeater.cc:511] Master 127.7.48.62:40725 requested a full tablet report, sending...
I20260928 06:46:06.502313 16648 ts_manager.cc:194] Registered new tserver with Master: be2c659f7158470bba0c9b60bb5162f1 (127.7.48.3:37831)
I20260928 06:46:06.502859 16648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.48.3:57413
I20260928 06:46:06.509492 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 06:46:06.512248 16687 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:46:06.511253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54682 (local address 127.7.48.62:40725)
0928 06:46:06.511380 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:06.511384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:06.511562 (+ 178us) server_negotiation.cc:407] Connection header received
0928 06:46:06.511714 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:06.511717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:06.511764 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:06.511859 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:06.512124 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54682: BlockingRecv error: recv got EOF from 127.0.0.1:54682 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:46:06.512353 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 16708
I20260928 06:46:06.519126 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 16839
I20260928 06:46:06.525979 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 16970
I20260928 06:46:06.531765 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 16617
2026-09-28T06:46:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (934 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T06:46:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:46:06Z Disabled control of system clock
I20260928 06:46:06.561228 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:36395
--webserver_interface=127.7.48.62
--webserver_port=0
--builtin_ntp_servers=127.7.48.20:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:36395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 06:46:06.668416 17109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:46:06.668685 17109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:46:06.668722 17109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:46:06.672675 17109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:46:06.672727 17109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:46:06.672742 17109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:46:06.672758 17109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:46:06.672773 17109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:46:06.677518 17109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:36395
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:36395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:46:06.678602 17109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:46:06.680006 17109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:46:06.688359 17115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:06.688367 17114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:06.688367 17117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:46:06.688817 17109 server_base.cc:1034] running on GCE node
I20260928 06:46:06.689342 17109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:46:06.690354 17109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:46:06.691504 17109 hybrid_clock.cc:648] HybridClock initialized: now 1790577966691476 us; error 43 us; skew 500 ppm
I20260928 06:46:06.693476 17109 webserver.cc:460] Webserver started at http://127.7.48.62:34237/ using document root <none> and password file <none>
I20260928 06:46:06.694065 17109 fs_manager.cc:359] Metadata directory not provided
I20260928 06:46:06.694131 17109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:46:06.694401 17109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:46:06.696512 17109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "662c42856dbd4b42968f9f82520178be"
format_stamp: "Formatted at 2026-09-28 06:46:06 on dist-test-slave-twl6"
server_key: "07202a3540f7cad980815b05c9d5254c"
server_key_iv: "f984b283a37cb712cdf05ba32d591c65"
server_key_version: "encryptionkey@0"
I20260928 06:46:06.697079 17109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "662c42856dbd4b42968f9f82520178be"
format_stamp: "Formatted at 2026-09-28 06:46:06 on dist-test-slave-twl6"
server_key: "07202a3540f7cad980815b05c9d5254c"
server_key_iv: "f984b283a37cb712cdf05ba32d591c65"
server_key_version: "encryptionkey@0"
I20260928 06:46:06.700822 17109 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:46:06.703233 17123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:06.704540 17109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:46:06.704672 17109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "662c42856dbd4b42968f9f82520178be"
format_stamp: "Formatted at 2026-09-28 06:46:06 on dist-test-slave-twl6"
server_key: "07202a3540f7cad980815b05c9d5254c"
server_key_iv: "f984b283a37cb712cdf05ba32d591c65"
server_key_version: "encryptionkey@0"
I20260928 06:46:06.704777 17109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:46:06.709384 17109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:46:06.709998 17109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:46:06.710191 17109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:46:06.717126 17109 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:36395
I20260928 06:46:06.717135 17175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:36395 every 8 connection(s)
I20260928 06:46:06.718113 17109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:46:06.721436 17176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:46:06.727010 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 17109
I20260928 06:46:06.727142 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:46:06.727180 17176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 662c42856dbd4b42968f9f82520178be: Bootstrap starting.
I20260928 06:46:06.727430 7360 external_mini_cluster.cc:1428] Setting key 2d0a001f6adde0f3aaab712fe3ff0f66
I20260928 06:46:06.729899 17176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 662c42856dbd4b42968f9f82520178be: Neither blocks nor log segments found. Creating new log.
I20260928 06:46:06.730657 17176 log.cc:824] T 00000000000000000000000000000000 P 662c42856dbd4b42968f9f82520178be: Log is configured to *not* fsync() on all Append() calls
I20260928 06:46:06.732841 17176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 662c42856dbd4b42968f9f82520178be: No bootstrap required, opened a new log
I20260928 06:46:06.733217 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:06.728842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54330 (local address 127.7.48.62:36395)
0928 06:46:06.729468 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:06.729484 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:06.729519 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:46:06.730259 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:06.730279 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:06.730649 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:06.730994 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:06.731827 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:46:06.731832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:46:06.731844 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:46:06.731853 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:46:06.731975 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:46:06.731994 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:46:06.732111 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:06.732240 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 06:46:06.732480 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":154,"threads_started":1}
I20260928 06:46:06.736821 17176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 662c42856dbd4b42968f9f82520178be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662c42856dbd4b42968f9f82520178be" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36395 } }
I20260928 06:46:06.737035 17176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 662c42856dbd4b42968f9f82520178be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:46:06.737124 17176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 662c42856dbd4b42968f9f82520178be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662c42856dbd4b42968f9f82520178be, State: Initialized, Role: FOLLOWER
I20260928 06:46:06.737581 17176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 662c42856dbd4b42968f9f82520178be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662c42856dbd4b42968f9f82520178be" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36395 } }
I20260928 06:46:06.737723 17176 raft_consensus.cc:388] T 00000000000000000000000000000000 P 662c42856dbd4b42968f9f82520178be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:46:06.737783 17176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 662c42856dbd4b42968f9f82520178be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:46:06.737900 17176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 662c42856dbd4b42968f9f82520178be [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:46:06.738881 17176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 662c42856dbd4b42968f9f82520178be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662c42856dbd4b42968f9f82520178be" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36395 } }
I20260928 06:46:06.739216 17176 leader_election.cc:302] T 00000000000000000000000000000000 P 662c42856dbd4b42968f9f82520178be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 662c42856dbd4b42968f9f82520178be; no voters:
I20260928 06:46:06.739531 17176 leader_election.cc:288] T 00000000000000000000000000000000 P 662c42856dbd4b42968f9f82520178be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:46:06.739708 17181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 662c42856dbd4b42968f9f82520178be [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:46:06.740062 17181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 662c42856dbd4b42968f9f82520178be [term 1 LEADER]: Becoming Leader. State: Replica: 662c42856dbd4b42968f9f82520178be, State: Running, Role: LEADER
I20260928 06:46:06.740377 17181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 662c42856dbd4b42968f9f82520178be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662c42856dbd4b42968f9f82520178be" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36395 } }
I20260928 06:46:06.741019 17176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 662c42856dbd4b42968f9f82520178be [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:46:06.742306 17183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 662c42856dbd4b42968f9f82520178be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 662c42856dbd4b42968f9f82520178be. Latest consensus state: current_term: 1 leader_uuid: "662c42856dbd4b42968f9f82520178be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662c42856dbd4b42968f9f82520178be" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36395 } } }
I20260928 06:46:06.742311 17182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 662c42856dbd4b42968f9f82520178be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "662c42856dbd4b42968f9f82520178be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662c42856dbd4b42968f9f82520178be" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36395 } } }
I20260928 06:46:06.742553 17182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 662c42856dbd4b42968f9f82520178be [sys.catalog]: This master's current role is: LEADER
I20260928 06:46:06.742496 17183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 662c42856dbd4b42968f9f82520178be [sys.catalog]: This master's current role is: LEADER
I20260928 06:46:06.742931 17191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:46:06.745710 17191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:46:06.750851 17191 catalog_manager.cc:1347] Generated new cluster ID: b4b6a9fc41844dbb828a461adaae8233
I20260928 06:46:06.750922 17191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:46:06.768537 17191 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:46:06.768882 17191 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:46:06.781229 17191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 662c42856dbd4b42968f9f82520178be: Generated new TSK 0
I20260928 06:46:06.782114 17191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:46:06.791471 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:36395
--builtin_ntp_servers=127.7.48.20:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 06:46:06.905014 17200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:46:06.905275 17200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:46:06.905315 17200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:46:06.909639 17200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:46:06.909698 17200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:46:06.909797 17200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:46:06.914587 17200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:36395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:46:06.915791 17200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:46:06.917373 17200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:46:06.927289 17208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:06.927297 17206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:06.927297 17205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:46:06.927618 17200 server_base.cc:1034] running on GCE node
I20260928 06:46:06.928102 17200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:46:06.928742 17200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:46:06.929917 17200 hybrid_clock.cc:648] HybridClock initialized: now 1790577966929901 us; error 28 us; skew 500 ppm
I20260928 06:46:06.931985 17200 webserver.cc:460] Webserver started at http://127.7.48.1:33611/ using document root <none> and password file <none>
I20260928 06:46:06.932595 17200 fs_manager.cc:359] Metadata directory not provided
I20260928 06:46:06.932660 17200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:46:06.932919 17200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:46:06.934986 17200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "00005f64774642c3af00d52bbf6cdb8d"
format_stamp: "Formatted at 2026-09-28 06:46:06 on dist-test-slave-twl6"
server_key: "52ad4ae22ca05a1845ded1a82bb73a1d"
server_key_iv: "26184c71b4c97c76b07480dcacb3b8e7"
server_key_version: "encryptionkey@0"
I20260928 06:46:06.935582 17200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "00005f64774642c3af00d52bbf6cdb8d"
format_stamp: "Formatted at 2026-09-28 06:46:06 on dist-test-slave-twl6"
server_key: "52ad4ae22ca05a1845ded1a82bb73a1d"
server_key_iv: "26184c71b4c97c76b07480dcacb3b8e7"
server_key_version: "encryptionkey@0"
I20260928 06:46:06.939280 17200 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 06:46:06.942102 17214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:06.943555 17200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:46:06.943691 17200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "00005f64774642c3af00d52bbf6cdb8d"
format_stamp: "Formatted at 2026-09-28 06:46:06 on dist-test-slave-twl6"
server_key: "52ad4ae22ca05a1845ded1a82bb73a1d"
server_key_iv: "26184c71b4c97c76b07480dcacb3b8e7"
server_key_version: "encryptionkey@0"
I20260928 06:46:06.943813 17200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:46:06.948510 17200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:46:06.949126 17200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:46:06.949313 17200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:46:06.949872 17200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:46:06.950780 17200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:46:06.950850 17200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:06.950917 17200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:46:06.950968 17200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:06.960187 17200 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:35313
I20260928 06:46:06.960223 17327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:35313 every 8 connection(s)
I20260928 06:46:06.961115 17200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:46:06.967516 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:06.963071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:60923 (local address 127.7.48.62:36395)
0928 06:46:06.963222 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:06.963225 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:06.964411 (+ 1186us) server_negotiation.cc:407] Connection header received
0928 06:46:06.965768 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:06.965772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:06.965816 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:06.965901 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:06.966997 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:46:06.967001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:46:06.967004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:46:06.967016 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:46:06.967040 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:46:06.967051 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:46:06.967159 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:06.967310 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 06:46:06.967371 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:46:06.968055 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 17200
I20260928 06:46:06.968149 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
I20260928 06:46:06.968425 7360 external_mini_cluster.cc:1428] Setting key 788760c8068a70326ff4fb82019d1037
I20260928 06:46:06.968554 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:06.963468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:36395 (local address 127.7.48.1:60923)
0928 06:46:06.964220 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:06.964276 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:06.965537 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:06.966070 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:06.966077 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:06.966168 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:06.966730 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:46:06.966732 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:06.966773 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:46:06.966841 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:46:06.967142 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:46:06.967151 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:46:06.967156 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:46:06.967342 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 06:46:06.967595 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":113,"threads_started":1}
I20260928 06:46:06.969889 17328 heartbeater.cc:347] Connected to a master server at 127.7.48.62:36395
I20260928 06:46:06.970120 17328 heartbeater.cc:464] Registering TS with master...
I20260928 06:46:06.970479 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:36395
--builtin_ntp_servers=127.7.48.20:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 06:46:06.970526 17328 heartbeater.cc:511] Master 127.7.48.62:36395 requested a full tablet report, sending...
I20260928 06:46:06.972196 17140 ts_manager.cc:194] Registered new tserver with Master: 00005f64774642c3af00d52bbf6cdb8d (127.7.48.1:35313)
W20260928 06:46:07.081753 17331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:46:07.082036 17331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:46:07.082079 17331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:46:07.086422 17331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:46:07.086483 17331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:46:07.086544 17331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:46:07.091487 17331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:36395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:46:07.092732 17331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:46:07.094236 17331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:46:07.104318 17336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:07.104295 17339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:46:07.104355 17331 server_base.cc:1034] running on GCE node
W20260928 06:46:07.104297 17337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:46:07.105100 17331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:46:07.105764 17331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:46:07.106966 17331 hybrid_clock.cc:648] HybridClock initialized: now 1790577967106958 us; error 37 us; skew 500 ppm
I20260928 06:46:07.108940 17331 webserver.cc:460] Webserver started at http://127.7.48.2:38167/ using document root <none> and password file <none>
I20260928 06:46:07.109615 17331 fs_manager.cc:359] Metadata directory not provided
I20260928 06:46:07.109710 17331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:46:07.109948 17331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:46:07.112149 17331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "3f455ef51aed42558d6f3e0e8fbc80b1"
format_stamp: "Formatted at 2026-09-28 06:46:07 on dist-test-slave-twl6"
server_key: "f00d5d51371769f9a2acdf9ee82c534b"
server_key_iv: "a6ff400769c7a668ac42f15f2875bab3"
server_key_version: "encryptionkey@0"
I20260928 06:46:07.112731 17331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "3f455ef51aed42558d6f3e0e8fbc80b1"
format_stamp: "Formatted at 2026-09-28 06:46:07 on dist-test-slave-twl6"
server_key: "f00d5d51371769f9a2acdf9ee82c534b"
server_key_iv: "a6ff400769c7a668ac42f15f2875bab3"
server_key_version: "encryptionkey@0"
I20260928 06:46:07.116605 17331 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 06:46:07.119123 17345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:07.120366 17331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 06:46:07.120509 17331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "3f455ef51aed42558d6f3e0e8fbc80b1"
format_stamp: "Formatted at 2026-09-28 06:46:07 on dist-test-slave-twl6"
server_key: "f00d5d51371769f9a2acdf9ee82c534b"
server_key_iv: "a6ff400769c7a668ac42f15f2875bab3"
server_key_version: "encryptionkey@0"
I20260928 06:46:07.120627 17331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:46:07.125171 17331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:46:07.125782 17331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:46:07.125964 17331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:46:07.126577 17331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:46:07.127476 17331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:46:07.127547 17331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:07.127614 17331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:46:07.127661 17331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:07.136749 17331 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:33329
I20260928 06:46:07.136772 17458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:33329 every 8 connection(s)
I20260928 06:46:07.137667 17331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:46:07.144048 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:07.139506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:47961 (local address 127.7.48.62:36395)
0928 06:46:07.139651 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:07.139655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:07.140747 (+ 1092us) server_negotiation.cc:407] Connection header received
0928 06:46:07.142301 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:07.142303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:07.142342 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:07.142418 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:07.143616 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:46:07.143619 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:46:07.143621 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:46:07.143631 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:46:07.143648 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:46:07.143655 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:46:07.143748 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:07.143842 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 06:46:07.143920 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:46:07.144948 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:07.139849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:36395 (local address 127.7.48.2:47961)
0928 06:46:07.140589 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:07.140630 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:07.142088 (+ 1458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:07.142588 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:07.142596 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:07.142683 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:07.143346 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:46:07.143348 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:07.143390 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:46:07.143462 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:46:07.143733 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:46:07.143738 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:46:07.143741 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:46:07.143861 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 06:46:07.144121 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":114,"threads_started":1}
I20260928 06:46:07.146014 17459 heartbeater.cc:347] Connected to a master server at 127.7.48.62:36395
I20260928 06:46:07.146251 17459 heartbeater.cc:464] Registering TS with master...
I20260928 06:46:07.146675 17459 heartbeater.cc:511] Master 127.7.48.62:36395 requested a full tablet report, sending...
I20260928 06:46:07.146978 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 17331
I20260928 06:46:07.147104 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:46:07.147411 7360 external_mini_cluster.cc:1428] Setting key da27777b1d3d43d38886f5b4c2067961
I20260928 06:46:07.147948 17140 ts_manager.cc:194] Registered new tserver with Master: 3f455ef51aed42558d6f3e0e8fbc80b1 (127.7.48.2:33329)
I20260928 06:46:07.149480 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:36395
--builtin_ntp_servers=127.7.48.20:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 06:46:07.264161 17462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:46:07.264454 17462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:46:07.264503 17462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:46:07.268667 17462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:46:07.268733 17462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:46:07.268826 17462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:46:07.273707 17462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:36395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:46:07.274894 17462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:46:07.276388 17462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:46:07.286048 17470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:07.286022 17468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:07.286036 17467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:46:07.286371 17462 server_base.cc:1034] running on GCE node
I20260928 06:46:07.286861 17462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:46:07.287530 17462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:46:07.288717 17462 hybrid_clock.cc:648] HybridClock initialized: now 1790577967288708 us; error 30 us; skew 500 ppm
I20260928 06:46:07.290522 17462 webserver.cc:460] Webserver started at http://127.7.48.3:35549/ using document root <none> and password file <none>
I20260928 06:46:07.291122 17462 fs_manager.cc:359] Metadata directory not provided
I20260928 06:46:07.291213 17462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:46:07.291451 17462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:46:07.293604 17462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "bf3db8c917db4f79bbfa4226000bdb52"
format_stamp: "Formatted at 2026-09-28 06:46:07 on dist-test-slave-twl6"
server_key: "3c4695b5a2f2ff687272774142a7c244"
server_key_iv: "f304a32a8df6d78402f7e011f943d506"
server_key_version: "encryptionkey@0"
I20260928 06:46:07.294169 17462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "bf3db8c917db4f79bbfa4226000bdb52"
format_stamp: "Formatted at 2026-09-28 06:46:07 on dist-test-slave-twl6"
server_key: "3c4695b5a2f2ff687272774142a7c244"
server_key_iv: "f304a32a8df6d78402f7e011f943d506"
server_key_version: "encryptionkey@0"
I20260928 06:46:07.297968 17462 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 06:46:07.300554 17476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:07.301764 17462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 06:46:07.301905 17462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "bf3db8c917db4f79bbfa4226000bdb52"
format_stamp: "Formatted at 2026-09-28 06:46:07 on dist-test-slave-twl6"
server_key: "3c4695b5a2f2ff687272774142a7c244"
server_key_iv: "f304a32a8df6d78402f7e011f943d506"
server_key_version: "encryptionkey@0"
I20260928 06:46:07.302023 17462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:46:07.306734 17462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:46:07.307335 17462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:46:07.307512 17462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:46:07.308180 17462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:46:07.309201 17462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:46:07.309263 17462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:07.309330 17462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:46:07.309372 17462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:07.318017 17462 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:41367
I20260928 06:46:07.318078 17589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:41367 every 8 connection(s)
I20260928 06:46:07.318946 17462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:46:07.325287 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:07.320873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:48931 (local address 127.7.48.62:36395)
0928 06:46:07.321017 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:07.321021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:07.322073 (+ 1052us) server_negotiation.cc:407] Connection header received
0928 06:46:07.323478 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:07.323481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:07.323527 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:07.323659 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:07.324804 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:46:07.324807 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:46:07.324810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:46:07.324822 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:46:07.324848 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:46:07.324856 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:46:07.324951 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:07.325108 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 06:46:07.325153 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 06:46:07.325809 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 17462
I20260928 06:46:07.325924 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:46:07.326195 7360 external_mini_cluster.cc:1428] Setting key 166cbf9f88d8d54258585d6b688de86e
I20260928 06:46:07.326215 17592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:07.321279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:36395 (local address 127.7.48.3:48931)
0928 06:46:07.321926 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:07.321965 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:07.323228 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:07.323828 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:07.323835 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:07.323996 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:07.324554 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:46:07.324556 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:07.324598 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:46:07.324662 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:46:07.324946 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:46:07.324955 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:46:07.324963 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 06:46:07.325146 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 06:46:07.325382 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":116,"threads_started":1}
I20260928 06:46:07.327366 17590 heartbeater.cc:347] Connected to a master server at 127.7.48.62:36395
I20260928 06:46:07.327580 17590 heartbeater.cc:464] Registering TS with master...
I20260928 06:46:07.328048 17590 heartbeater.cc:511] Master 127.7.48.62:36395 requested a full tablet report, sending...
I20260928 06:46:07.329026 17140 ts_manager.cc:194] Registered new tserver with Master: bf3db8c917db4f79bbfa4226000bdb52 (127.7.48.3:41367)
I20260928 06:46:07.339916 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 06:46:07.343032 17179 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:46:07.341902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54336 (local address 127.7.48.62:36395)
0928 06:46:07.342049 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:07.342053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:07.342425 (+ 372us) server_negotiation.cc:407] Connection header received
0928 06:46:07.342545 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:07.342548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:07.342589 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:07.342677 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:07.342905 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54336: BlockingRecv error: recv got EOF from 127.0.0.1:54336 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:46:07.343173 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 17200
I20260928 06:46:07.348979 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 17331
I20260928 06:46:07.354606 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 17462
I20260928 06:46:07.359925 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 17109
2026-09-28T06:46:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (827 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:46:07 dist-test-slave-twl6 krb5kdc[17600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:46:07 dist-test-slave-twl6 krb5kdc[17600](info): set up 2 sockets
Sep 28 06:46:07 dist-test-slave-twl6 krb5kdc[17600](info): commencing operation
krb5kdc: starting...
W20260928 06:46:08.393575 7360 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:46:08 dist-test-slave-twl6 krb5kdc[17600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:46:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:46:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:46:08.515156 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:32821
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:32821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:46:08.629333 17613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:46:08.629693 17613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:46:08.629770 17613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:46:08.633988 17613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:46:08.634093 17613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:46:08.634131 17613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:46:08.634171 17613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:46:08.634207 17613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:46:08.639621 17613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:32821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:32821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:46:08.640847 17613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:46:08.641932 17613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:46:08.650674 17618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:08.650660 17621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:08.650745 17619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:46:08.650959 17613 server_base.cc:1034] running on GCE node
I20260928 06:46:08.651527 17613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:46:08.652750 17613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:46:08.653913 17613 hybrid_clock.cc:648] HybridClock initialized: now 1790577968653883 us; error 43 us; skew 500 ppm
Sep 28 06:46:08 dist-test-slave-twl6 krb5kdc[17600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577968, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:46:08.656592 17613 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:46:08.657473 17613 webserver.cc:460] Webserver started at http://127.7.48.62:43753/ using document root <none> and password file <none>
I20260928 06:46:08.658125 17613 fs_manager.cc:359] Metadata directory not provided
I20260928 06:46:08.658226 17613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:46:08.658469 17613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:46:08.660660 17613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "9d31689ac2384b3d87933dcce2f0169b"
format_stamp: "Formatted at 2026-09-28 06:46:08 on dist-test-slave-twl6"
server_key: "a13b27735b4eaa6f4d194786b623fea8"
server_key_iv: "4d78a174c507a5151e5f206179c405d9"
server_key_version: "encryptionkey@0"
I20260928 06:46:08.661250 17613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "9d31689ac2384b3d87933dcce2f0169b"
format_stamp: "Formatted at 2026-09-28 06:46:08 on dist-test-slave-twl6"
server_key: "a13b27735b4eaa6f4d194786b623fea8"
server_key_iv: "4d78a174c507a5151e5f206179c405d9"
server_key_version: "encryptionkey@0"
I20260928 06:46:08.665079 17613 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 06:46:08.667446 17628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:08.668848 17613 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:46:08.669030 17613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "9d31689ac2384b3d87933dcce2f0169b"
format_stamp: "Formatted at 2026-09-28 06:46:08 on dist-test-slave-twl6"
server_key: "a13b27735b4eaa6f4d194786b623fea8"
server_key_iv: "4d78a174c507a5151e5f206179c405d9"
server_key_version: "encryptionkey@0"
I20260928 06:46:08.669150 17613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:46:08.681481 17613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:46:08.684230 17613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:46:08.684428 17613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:46:08.691283 17613 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:32821
I20260928 06:46:08.691581 17680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:32821 every 8 connection(s)
I20260928 06:46:08.692394 17613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:46:08.695272 17681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:46:08.700950 17681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d31689ac2384b3d87933dcce2f0169b: Bootstrap starting.
I20260928 06:46:08.701850 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 17613
I20260928 06:46:08.701956 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:46:08.702255 7360 external_mini_cluster.cc:1428] Setting key 8b110d597164804567336dac9c09d482
I20260928 06:46:08.703867 17681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d31689ac2384b3d87933dcce2f0169b: Neither blocks nor log segments found. Creating new log.
I20260928 06:46:08.704684 17681 log.cc:824] T 00000000000000000000000000000000 P 9d31689ac2384b3d87933dcce2f0169b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:46:08.707002 17681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d31689ac2384b3d87933dcce2f0169b: No bootstrap required, opened a new log
I20260928 06:46:08.711050 17681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d31689ac2384b3d87933dcce2f0169b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d31689ac2384b3d87933dcce2f0169b" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 32821 } }
I20260928 06:46:08.711306 17681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d31689ac2384b3d87933dcce2f0169b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:46:08.711400 17681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d31689ac2384b3d87933dcce2f0169b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d31689ac2384b3d87933dcce2f0169b, State: Initialized, Role: FOLLOWER
Sep 28 06:46:08 dist-test-slave-twl6 krb5kdc[17600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:46:08.711962 17681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d31689ac2384b3d87933dcce2f0169b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d31689ac2384b3d87933dcce2f0169b" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 32821 } }
I20260928 06:46:08.712132 17681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d31689ac2384b3d87933dcce2f0169b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:46:08.712203 17681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d31689ac2384b3d87933dcce2f0169b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:46:08.712306 17681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d31689ac2384b3d87933dcce2f0169b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:46:08.713258 17681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d31689ac2384b3d87933dcce2f0169b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d31689ac2384b3d87933dcce2f0169b" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 32821 } }
I20260928 06:46:08.713629 17681 leader_election.cc:302] T 00000000000000000000000000000000 P 9d31689ac2384b3d87933dcce2f0169b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d31689ac2384b3d87933dcce2f0169b; no voters:
I20260928 06:46:08.714008 17681 leader_election.cc:288] T 00000000000000000000000000000000 P 9d31689ac2384b3d87933dcce2f0169b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:46:08.714095 17686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d31689ac2384b3d87933dcce2f0169b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:46:08.714321 17686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d31689ac2384b3d87933dcce2f0169b [term 1 LEADER]: Becoming Leader. State: Replica: 9d31689ac2384b3d87933dcce2f0169b, State: Running, Role: LEADER
I20260928 06:46:08.714689 17686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d31689ac2384b3d87933dcce2f0169b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d31689ac2384b3d87933dcce2f0169b" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 32821 } }
I20260928 06:46:08.715369 17681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d31689ac2384b3d87933dcce2f0169b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:46:08.715965 17688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d31689ac2384b3d87933dcce2f0169b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d31689ac2384b3d87933dcce2f0169b. Latest consensus state: current_term: 1 leader_uuid: "9d31689ac2384b3d87933dcce2f0169b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d31689ac2384b3d87933dcce2f0169b" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 32821 } } }
I20260928 06:46:08.716086 17688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d31689ac2384b3d87933dcce2f0169b [sys.catalog]: This master's current role is: LEADER
I20260928 06:46:08.716806 17687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d31689ac2384b3d87933dcce2f0169b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d31689ac2384b3d87933dcce2f0169b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d31689ac2384b3d87933dcce2f0169b" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 32821 } } }
I20260928 06:46:08.716962 17687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d31689ac2384b3d87933dcce2f0169b [sys.catalog]: This master's current role is: LEADER
I20260928 06:46:08.718477 17684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:08.703464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37796 (local address 127.7.48.62:32821)
0928 06:46:08.703977 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:08.703995 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:08.704056 (+ 61us) server_negotiation.cc:407] Connection header received
0928 06:46:08.704734 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:08.704752 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:08.705144 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:08.705431 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:08.706171 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:08.707674 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:08.708568 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:08.709285 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:08.709470 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:08.712235 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:46:08.712243 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:46:08.712263 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:46:08.712300 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:46:08.714407 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:46:08.714923 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:46:08.714934 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:46:08.714939 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:46:08.715051 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:46:08.715370 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:46:08.715373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:46:08.715375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:46:08.715956 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:46:08.716143 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:08.717044 (+ 901us) server_negotiation.cc:299] Negotiation successful
0928 06:46:08.717349 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":160,"threads_started":1}
W20260928 06:46:08.721057 17702 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d31689ac2384b3d87933dcce2f0169b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:46:08.721153 17702 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:46:08.721262 17691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:46:08.721997 17691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:46:08.727949 17691 catalog_manager.cc:1347] Generated new cluster ID: d423c279f7a4483eb1140d9e4e405cc0
I20260928 06:46:08.728024 17691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:46:08.739267 17691 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:46:08.740288 17691 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:46:08.744277 17691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d31689ac2384b3d87933dcce2f0169b: Generated new TSK 0
I20260928 06:46:08.745085 17691 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
I20260928 06:46:08.779220 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:32821
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:46:08.890524 17707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:46:08.890836 17707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:46:08.890910 17707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:46:08.894932 17707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:46:08.895028 17707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:46:08.895128 17707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:46:08.900393 17707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:32821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:46:08.901540 17707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:46:08.902643 17707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:46:08.912463 17713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:08.912479 17715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:08.912672 17712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:46:08.912679 17707 server_base.cc:1034] running on GCE node
I20260928 06:46:08.913280 17707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:46:08.914008 17707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:46:08.915205 17707 hybrid_clock.cc:648] HybridClock initialized: now 1790577968915185 us; error 41 us; skew 500 ppm
Sep 28 06:46:08 dist-test-slave-twl6 krb5kdc[17600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577968, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:46:08.918097 17707 init.cc:377] Logged in from keytab as kudu/127.7.48.1@KRBTEST.COM (short username kudu)
I20260928 06:46:08.918999 17707 webserver.cc:460] Webserver started at http://127.7.48.1:43171/ using document root <none> and password file <none>
I20260928 06:46:08.919526 17707 fs_manager.cc:359] Metadata directory not provided
I20260928 06:46:08.919586 17707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:46:08.919749 17707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:46:08.921777 17707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "80ecadd1915b41cc86a82c7c56004884"
format_stamp: "Formatted at 2026-09-28 06:46:08 on dist-test-slave-twl6"
server_key: "0746d597eb890d292ff3d29642d582e4"
server_key_iv: "5735436ee7392a517722960d93a1212e"
server_key_version: "encryptionkey@0"
I20260928 06:46:08.922299 17707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "80ecadd1915b41cc86a82c7c56004884"
format_stamp: "Formatted at 2026-09-28 06:46:08 on dist-test-slave-twl6"
server_key: "0746d597eb890d292ff3d29642d582e4"
server_key_iv: "5735436ee7392a517722960d93a1212e"
server_key_version: "encryptionkey@0"
I20260928 06:46:08.925942 17707 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 06:46:08.928371 17722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:08.929584 17707 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:46:08.929680 17707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "80ecadd1915b41cc86a82c7c56004884"
format_stamp: "Formatted at 2026-09-28 06:46:08 on dist-test-slave-twl6"
server_key: "0746d597eb890d292ff3d29642d582e4"
server_key_iv: "5735436ee7392a517722960d93a1212e"
server_key_version: "encryptionkey@0"
I20260928 06:46:08.929761 17707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:46:08.945513 17707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:46:08.948807 17707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:46:08.948958 17707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:46:08.949486 17707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:46:08.950253 17707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:46:08.950297 17707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:08.950335 17707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:46:08.950354 17707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:08.958812 17707 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:36543
I20260928 06:46:08.958901 17835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:36543 every 8 connection(s)
I20260928 06:46:08.959709 17707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:46:08.965905 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 17707
I20260928 06:46:08.966012 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
I20260928 06:46:08.966281 7360 external_mini_cluster.cc:1428] Setting key 2d6cffbdc1a3270305d9f8bc68ffa8ce
Sep 28 06:46:08 dist-test-slave-twl6 krb5kdc[17600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577968, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:46:08.973517 17684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:08.961473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:48659 (local address 127.7.48.62:32821)
0928 06:46:08.961626 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:08.961629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:08.962408 (+ 779us) server_negotiation.cc:407] Connection header received
0928 06:46:08.963726 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:08.963730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:08.963787 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:08.963870 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:08.965052 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:08.965591 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:08.967018 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:08.967270 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:08.967380 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:08.969870 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:46:08.969874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:46:08.969877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:46:08.969907 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:46:08.971406 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:46:08.971918 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:46:08.971921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:46:08.971923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:46:08.972003 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:46:08.972346 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:46:08.972367 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:46:08.972369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:46:08.972550 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:46:08.972710 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:08.973214 (+ 504us) server_negotiation.cc:299] Negotiation successful
0928 06:46:08.973333 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:46:08.974351 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:08.961829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:32821 (local address 127.7.48.1:48659)
0928 06:46:08.962255 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:08.962297 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:08.963510 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:08.964085 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:08.964092 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:08.964427 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:08.964894 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:08.964906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:08.965720 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:08.965726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:08.966896 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:08.966906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:08.967430 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:08.967434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:08.967659 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:08.968315 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:46:08.968331 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:08.969712 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:46:08.971515 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:46:08.971520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:46:08.971535 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:46:08.971788 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:46:08.972094 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:46:08.972098 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:46:08.972100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:46:08.972251 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:46:08.972710 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:46:08.972718 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:46:08.972977 (+ 259us) client_negotiation.cc:769] Sending connection context
0928 06:46:08.973161 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 06:46:08.973497 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.7.48.2@KRBTEST.COM; defaulting to no policy
I20260928 06:46:08.975566 17836 heartbeater.cc:347] Connected to a master server at 127.7.48.62:32821
I20260928 06:46:08.975802 17836 heartbeater.cc:464] Registering TS with master...
I20260928 06:46:08.976292 17836 heartbeater.cc:511] Master 127.7.48.62:32821 requested a full tablet report, sending...
I20260928 06:46:08.977847 17645 ts_manager.cc:194] Registered new tserver with Master: 80ecadd1915b41cc86a82c7c56004884 (127.7.48.1:36543)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.2@KRBTEST.COM" created.
I20260928 06:46:08.979454 17645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.1@KRBTEST.COM'} at 127.7.48.1:48659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
I20260928 06:46:08.994377 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:32821
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:46:09.104648 17841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:46:09.104944 17841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:46:09.105020 17841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:46:09.109265 17841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:46:09.109351 17841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:46:09.109439 17841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:46:09.114749 17841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:32821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:46:09.115998 17841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:46:09.117200 17841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:46:09.128782 17847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:09.129011 17846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:09.129309 17849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:46:09.129468 17841 server_base.cc:1034] running on GCE node
I20260928 06:46:09.129926 17841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:46:09.130599 17841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:46:09.131784 17841 hybrid_clock.cc:648] HybridClock initialized: now 1790577969131759 us; error 39 us; skew 500 ppm
Sep 28 06:46:09 dist-test-slave-twl6 krb5kdc[17600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577969, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:46:09.134729 17841 init.cc:377] Logged in from keytab as kudu/127.7.48.2@KRBTEST.COM (short username kudu)
I20260928 06:46:09.135596 17841 webserver.cc:460] Webserver started at http://127.7.48.2:37255/ using document root <none> and password file <none>
I20260928 06:46:09.136183 17841 fs_manager.cc:359] Metadata directory not provided
I20260928 06:46:09.136245 17841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:46:09.136428 17841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:46:09.138432 17841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "e22b2e87d56c406f9934d548ef018e33"
format_stamp: "Formatted at 2026-09-28 06:46:09 on dist-test-slave-twl6"
server_key: "e7a92647daeb820321131b202265bc01"
server_key_iv: "c663fa3cfa9d9ace9a592efdccd37723"
server_key_version: "encryptionkey@0"
I20260928 06:46:09.138931 17841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "e22b2e87d56c406f9934d548ef018e33"
format_stamp: "Formatted at 2026-09-28 06:46:09 on dist-test-slave-twl6"
server_key: "e7a92647daeb820321131b202265bc01"
server_key_iv: "c663fa3cfa9d9ace9a592efdccd37723"
server_key_version: "encryptionkey@0"
I20260928 06:46:09.142645 17841 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 06:46:09.145136 17856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:09.146402 17841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:46:09.146528 17841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "e22b2e87d56c406f9934d548ef018e33"
format_stamp: "Formatted at 2026-09-28 06:46:09 on dist-test-slave-twl6"
server_key: "e7a92647daeb820321131b202265bc01"
server_key_iv: "c663fa3cfa9d9ace9a592efdccd37723"
server_key_version: "encryptionkey@0"
I20260928 06:46:09.146680 17841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:46:09.166925 17841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:46:09.170403 17841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:46:09.170603 17841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:46:09.171226 17841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:46:09.172147 17841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:46:09.172194 17841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:09.172255 17841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:46:09.172290 17841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:09.181373 17841 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:33995
I20260928 06:46:09.181398 17969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:33995 every 8 connection(s)
I20260928 06:46:09.182421 17841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:46:09.190847 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 17841
I20260928 06:46:09.190979 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:46:09.191305 7360 external_mini_cluster.cc:1428] Setting key cd830c6df0c1a8290b39310a084f962b
Sep 28 06:46:09 dist-test-slave-twl6 krb5kdc[17600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577969, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:46:09.196691 17684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:09.184513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:41603 (local address 127.7.48.62:32821)
0928 06:46:09.184652 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:09.184656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:09.185503 (+ 847us) server_negotiation.cc:407] Connection header received
0928 06:46:09.186901 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:09.186904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:09.186959 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:09.187039 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:09.188307 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.188827 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.189989 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.190219 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.190331 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:09.193083 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:46:09.193086 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:46:09.193088 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:46:09.193111 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:46:09.194570 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:46:09.195092 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:46:09.195096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:46:09.195097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:46:09.195183 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:46:09.195544 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:46:09.195548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:46:09.195549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:46:09.195738 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:46:09.195843 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:09.196405 (+ 562us) server_negotiation.cc:299] Negotiation successful
0928 06:46:09.196529 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:46:09.197548 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:09.184915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:32821 (local address 127.7.48.2:41603)
0928 06:46:09.185353 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:09.185396 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:09.186689 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:09.187216 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:09.187223 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:09.187599 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:09.188158 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:09.188172 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.188954 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.188963 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:09.189861 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:09.189870 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.190373 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.190377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:09.190570 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:09.191240 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:46:09.191255 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:09.192902 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:46:09.194703 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:46:09.194709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:46:09.194724 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:46:09.194982 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:46:09.195272 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:46:09.195276 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:46:09.195278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:46:09.195438 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:46:09.195843 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:46:09.195851 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:46:09.196149 (+ 298us) client_negotiation.cc:769] Sending connection context
0928 06:46:09.196345 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 06:46:09.196690 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":121,"threads_started":1}
I20260928 06:46:09.198809 17970 heartbeater.cc:347] Connected to a master server at 127.7.48.62:32821
I20260928 06:46:09.199088 17970 heartbeater.cc:464] Registering TS with master...
I20260928 06:46:09.199743 17970 heartbeater.cc:511] Master 127.7.48.62:32821 requested a full tablet report, sending...
I20260928 06:46:09.200912 17645 ts_manager.cc:194] Registered new tserver with Master: e22b2e87d56c406f9934d548ef018e33 (127.7.48.2:33995)
WARNING: no policy specified for kudu/127.7.48.3@KRBTEST.COM; defaulting to no policy
I20260928 06:46:09.201514 17645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.2@KRBTEST.COM'} at 127.7.48.2:41603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
I20260928 06:46:09.217092 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:32821
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:46:09.328938 17975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:46:09.329213 17975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:46:09.329257 17975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:46:09.333292 17975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:46:09.333345 17975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:46:09.333402 17975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:46:09.338739 17975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:32821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:46:09.339720 17975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:46:09.340788 17975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:46:09.350666 17980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:09.350790 17981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:46:09.350908 17975 server_base.cc:1034] running on GCE node
W20260928 06:46:09.350965 17983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:46:09.351367 17975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:46:09.352077 17975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:46:09.353268 17975 hybrid_clock.cc:648] HybridClock initialized: now 1790577969353259 us; error 34 us; skew 500 ppm
Sep 28 06:46:09 dist-test-slave-twl6 krb5kdc[17600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577969, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:46:09.356041 17975 init.cc:377] Logged in from keytab as kudu/127.7.48.3@KRBTEST.COM (short username kudu)
I20260928 06:46:09.356935 17975 webserver.cc:460] Webserver started at http://127.7.48.3:37861/ using document root <none> and password file <none>
I20260928 06:46:09.357510 17975 fs_manager.cc:359] Metadata directory not provided
I20260928 06:46:09.357620 17975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:46:09.357832 17975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:46:09.360015 17975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "730a5444c89041b78ee2113d2ea2b47f"
format_stamp: "Formatted at 2026-09-28 06:46:09 on dist-test-slave-twl6"
server_key: "f7ecd6d6652660f5719367b75ad41817"
server_key_iv: "a8ad26bf5eeeda6594ef22d63b13ce7e"
server_key_version: "encryptionkey@0"
I20260928 06:46:09.360600 17975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "730a5444c89041b78ee2113d2ea2b47f"
format_stamp: "Formatted at 2026-09-28 06:46:09 on dist-test-slave-twl6"
server_key: "f7ecd6d6652660f5719367b75ad41817"
server_key_iv: "a8ad26bf5eeeda6594ef22d63b13ce7e"
server_key_version: "encryptionkey@0"
I20260928 06:46:09.364365 17975 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:46:09.366855 17990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:09.368144 17975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:46:09.368284 17975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "730a5444c89041b78ee2113d2ea2b47f"
format_stamp: "Formatted at 2026-09-28 06:46:09 on dist-test-slave-twl6"
server_key: "f7ecd6d6652660f5719367b75ad41817"
server_key_iv: "a8ad26bf5eeeda6594ef22d63b13ce7e"
server_key_version: "encryptionkey@0"
I20260928 06:46:09.368402 17975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:46:09.409945 17975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:46:09.413146 17975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:46:09.413396 17975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:46:09.414095 17975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:46:09.414960 17975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:46:09.415048 17975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:09.415104 17975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:46:09.415127 17975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:09.424165 17975 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:37379
I20260928 06:46:09.424211 18103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:37379 every 8 connection(s)
I20260928 06:46:09.425067 17975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:46:09.433903 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 17975
I20260928 06:46:09.434015 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:46:09.434386 7360 external_mini_cluster.cc:1428] Setting key ddc6fcfc4f0c4adf5bb94d9d70fe323d
Sep 28 06:46:09 dist-test-slave-twl6 krb5kdc[17600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577969, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:46:09.439147 17684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:09.426805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:43661 (local address 127.7.48.62:32821)
0928 06:46:09.426944 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:09.426948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:09.427739 (+ 791us) server_negotiation.cc:407] Connection header received
0928 06:46:09.429209 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:09.429213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:09.429275 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:09.429382 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:09.430567 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.431101 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.432318 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.432556 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.432677 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:09.435441 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:46:09.435445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:46:09.435450 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:46:09.435480 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:46:09.437107 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:46:09.437618 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:46:09.437622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:46:09.437624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:46:09.437708 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:46:09.438058 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:46:09.438062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:46:09.438064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:46:09.438248 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:46:09.438366 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:09.438855 (+ 489us) server_negotiation.cc:299] Negotiation successful
0928 06:46:09.438972 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:46:09.440060 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:09.427136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:32821 (local address 127.7.48.3:43661)
0928 06:46:09.427579 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:09.427624 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:09.428974 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:09.429561 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:09.429568 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:09.429915 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:09.430406 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:09.430419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.431231 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.431240 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:09.432191 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:09.432201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.432721 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.432725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:09.432895 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:09.433532 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:46:09.433548 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:09.435263 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:46:09.437232 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:46:09.437238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:46:09.437254 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:46:09.437512 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:46:09.437802 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:46:09.437806 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:46:09.437808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:46:09.437952 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:46:09.438366 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:46:09.438374 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:46:09.438625 (+ 251us) client_negotiation.cc:769] Sending connection context
0928 06:46:09.438813 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 06:46:09.439166 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":118,"threads_started":1}
I20260928 06:46:09.441154 18104 heartbeater.cc:347] Connected to a master server at 127.7.48.62:32821
I20260928 06:46:09.441373 18104 heartbeater.cc:464] Registering TS with master...
I20260928 06:46:09.441814 18104 heartbeater.cc:511] Master 127.7.48.62:32821 requested a full tablet report, sending...
I20260928 06:46:09.442787 17645 ts_manager.cc:194] Registered new tserver with Master: 730a5444c89041b78ee2113d2ea2b47f (127.7.48.3:37379)
I20260928 06:46:09.443305 17645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.3@KRBTEST.COM'} at 127.7.48.3:43661
I20260928 06:46:09.448349 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:46:09.457456 17684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:09.450277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37810 (local address 127.7.48.62:32821)
0928 06:46:09.450389 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:09.450394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:09.450544 (+ 150us) server_negotiation.cc:407] Connection header received
0928 06:46:09.450645 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:09.450647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:09.450683 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:09.450747 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:09.451532 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.452100 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.452893 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.453097 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.453173 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:09.454444 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:46:09.454447 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:46:09.454449 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:46:09.454471 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:46:09.456081 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:46:09.456459 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:46:09.456461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:46:09.456462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:46:09.456548 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:46:09.456771 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:46:09.456773 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:46:09.456775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:46:09.456973 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:46:09.457076 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:09.457247 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 06:46:09.457336 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:46:09.460307 17645 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:46:09.463457 17645 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:46:09.478403 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:09.472304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50346 (local address 127.7.48.2:33995)
0928 06:46:09.472620 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:09.472632 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:09.473181 (+ 549us) server_negotiation.cc:407] Connection header received
0928 06:46:09.473417 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:09.473423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:09.473665 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:09.473801 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:46:09.474296 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.475349 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.477097 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.477669 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.477797 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:09.477956 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:09.478142 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 06:46:09.478242 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":107,"threads_started":1}
I20260928 06:46:09.478389 18115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:09.471993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:33995 (local address 127.0.0.1:50346)
0928 06:46:09.473042 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:09.473092 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:09.473262 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:09.474067 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:09.474071 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:09.474092 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:46:09.474191 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:09.474196 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.475479 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.475481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:09.476944 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:09.476956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.477798 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.477801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:09.477917 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:09.477954 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 06:46:09.478107 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 06:46:09.478235 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":150,"thread_start_us":126,"threads_started":1}
I20260928 06:46:09.478777 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:09.471834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:37379 (local address 127.0.0.1:50478)
0928 06:46:09.473042 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:09.473094 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:09.473221 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:09.473877 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:09.473883 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:09.473906 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:46:09.474043 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:09.474053 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.475315 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.475320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:09.476944 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:09.476956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.478201 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.478205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:09.478341 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:09.478463 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 06:46:09.478539 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:46:09.478644 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"mutex_wait_us":168,"thread_start_us":161,"threads_started":1}
I20260928 06:46:09.478897 18120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:09.472823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57998 (local address 127.7.48.1:36543)
0928 06:46:09.473555 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:09.473572 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:09.473906 (+ 334us) server_negotiation.cc:407] Connection header received
0928 06:46:09.473985 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:09.473989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:09.474172 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:09.474312 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:46:09.474723 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.475722 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.477167 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.477910 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.477999 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:09.478135 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:09.478648 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 06:46:09.478742 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":120,"threads_started":1}
I20260928 06:46:09.479199 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:09.472632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:36543 (local address 127.0.0.1:57998)
0928 06:46:09.473706 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:09.473719 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:09.473812 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:09.474469 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:09.474472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:09.474489 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:46:09.474569 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:09.474575 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.476027 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.476030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:09.477002 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:09.477036 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.478201 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.478205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:09.478371 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:09.478466 (+ 95us) client_negotiation.cc:769] Sending connection context
0928 06:46:09.478541 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 06:46:09.479076 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1016,"mutex_wait_us":528,"thread_start_us":94,"threads_started":1}
I20260928 06:46:09.481676 17770 tablet_service.cc:1467] Processing CreateTablet for tablet 7ebac70a574647c3aaa0636caa0d50d9 (DEFAULT_TABLE table=test-table [id=688cd9b063a1412d99491a382371da94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:46:09.481698 17904 tablet_service.cc:1467] Processing CreateTablet for tablet 7ebac70a574647c3aaa0636caa0d50d9 (DEFAULT_TABLE table=test-table [id=688cd9b063a1412d99491a382371da94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:46:09.482678 17770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ebac70a574647c3aaa0636caa0d50d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:46:09.482681 17904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ebac70a574647c3aaa0636caa0d50d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:46:09.484359 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:09.472764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50478 (local address 127.7.48.3:37379)
0928 06:46:09.473221 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:09.473233 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:09.473255 (+ 22us) server_negotiation.cc:407] Connection header received
0928 06:46:09.473364 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:09.473369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:09.473686 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:09.473877 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:46:09.474163 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.475169 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.477078 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.477866 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.477983 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:09.478134 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:09.479213 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0928 06:46:09.484041 (+ 4828us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":118,"threads_started":1}
I20260928 06:46:09.487571 18038 tablet_service.cc:1467] Processing CreateTablet for tablet 7ebac70a574647c3aaa0636caa0d50d9 (DEFAULT_TABLE table=test-table [id=688cd9b063a1412d99491a382371da94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:46:09.488795 18038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ebac70a574647c3aaa0636caa0d50d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:46:09.489567 18121 tablet_bootstrap.cc:492] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884: Bootstrap starting.
I20260928 06:46:09.491842 18121 tablet_bootstrap.cc:654] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884: Neither blocks nor log segments found. Creating new log.
I20260928 06:46:09.492298 18122 tablet_bootstrap.cc:492] T 7ebac70a574647c3aaa0636caa0d50d9 P e22b2e87d56c406f9934d548ef018e33: Bootstrap starting.
I20260928 06:46:09.492805 18121 log.cc:824] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884: Log is configured to *not* fsync() on all Append() calls
I20260928 06:46:09.495090 18122 tablet_bootstrap.cc:654] T 7ebac70a574647c3aaa0636caa0d50d9 P e22b2e87d56c406f9934d548ef018e33: Neither blocks nor log segments found. Creating new log.
I20260928 06:46:09.495208 18121 tablet_bootstrap.cc:492] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884: No bootstrap required, opened a new log
I20260928 06:46:09.495467 18121 ts_tablet_manager.cc:1397] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 06:46:09.495968 18124 tablet_bootstrap.cc:492] T 7ebac70a574647c3aaa0636caa0d50d9 P 730a5444c89041b78ee2113d2ea2b47f: Bootstrap starting.
I20260928 06:46:09.495976 18122 log.cc:824] T 7ebac70a574647c3aaa0636caa0d50d9 P e22b2e87d56c406f9934d548ef018e33: Log is configured to *not* fsync() on all Append() calls
I20260928 06:46:09.498333 18124 tablet_bootstrap.cc:654] T 7ebac70a574647c3aaa0636caa0d50d9 P 730a5444c89041b78ee2113d2ea2b47f: Neither blocks nor log segments found. Creating new log.
I20260928 06:46:09.498468 18122 tablet_bootstrap.cc:492] T 7ebac70a574647c3aaa0636caa0d50d9 P e22b2e87d56c406f9934d548ef018e33: No bootstrap required, opened a new log
I20260928 06:46:09.498718 18122 ts_tablet_manager.cc:1397] T 7ebac70a574647c3aaa0636caa0d50d9 P e22b2e87d56c406f9934d548ef018e33: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 06:46:09.499729 18124 log.cc:824] T 7ebac70a574647c3aaa0636caa0d50d9 P 730a5444c89041b78ee2113d2ea2b47f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:46:09.500625 18121 raft_consensus.cc:348] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730a5444c89041b78ee2113d2ea2b47f" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 37379 } } peers { permanent_uuid: "e22b2e87d56c406f9934d548ef018e33" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33995 } } peers { permanent_uuid: "80ecadd1915b41cc86a82c7c56004884" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36543 } }
I20260928 06:46:09.500877 18121 raft_consensus.cc:374] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:46:09.500950 18121 raft_consensus.cc:729] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80ecadd1915b41cc86a82c7c56004884, State: Initialized, Role: FOLLOWER
I20260928 06:46:09.501433 18121 consensus_queue.cc:260] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730a5444c89041b78ee2113d2ea2b47f" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 37379 } } peers { permanent_uuid: "e22b2e87d56c406f9934d548ef018e33" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33995 } } peers { permanent_uuid: "80ecadd1915b41cc86a82c7c56004884" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36543 } }
I20260928 06:46:09.501852 18124 tablet_bootstrap.cc:492] T 7ebac70a574647c3aaa0636caa0d50d9 P 730a5444c89041b78ee2113d2ea2b47f: No bootstrap required, opened a new log
I20260928 06:46:09.502096 17836 heartbeater.cc:503] Master 127.7.48.62:32821 was elected leader, sending a full tablet report...
I20260928 06:46:09.502101 18124 ts_tablet_manager.cc:1397] T 7ebac70a574647c3aaa0636caa0d50d9 P 730a5444c89041b78ee2113d2ea2b47f: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260928 06:46:09.502477 18121 ts_tablet_manager.cc:1428] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260928 06:46:09.502681 18122 raft_consensus.cc:348] T 7ebac70a574647c3aaa0636caa0d50d9 P e22b2e87d56c406f9934d548ef018e33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730a5444c89041b78ee2113d2ea2b47f" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 37379 } } peers { permanent_uuid: "e22b2e87d56c406f9934d548ef018e33" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33995 } } peers { permanent_uuid: "80ecadd1915b41cc86a82c7c56004884" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36543 } }
I20260928 06:46:09.502950 18122 raft_consensus.cc:374] T 7ebac70a574647c3aaa0636caa0d50d9 P e22b2e87d56c406f9934d548ef018e33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:46:09.503052 18122 raft_consensus.cc:729] T 7ebac70a574647c3aaa0636caa0d50d9 P e22b2e87d56c406f9934d548ef018e33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e22b2e87d56c406f9934d548ef018e33, State: Initialized, Role: FOLLOWER
I20260928 06:46:09.503516 18122 consensus_queue.cc:260] T 7ebac70a574647c3aaa0636caa0d50d9 P e22b2e87d56c406f9934d548ef018e33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730a5444c89041b78ee2113d2ea2b47f" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 37379 } } peers { permanent_uuid: "e22b2e87d56c406f9934d548ef018e33" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33995 } } peers { permanent_uuid: "80ecadd1915b41cc86a82c7c56004884" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36543 } }
I20260928 06:46:09.504226 17970 heartbeater.cc:503] Master 127.7.48.62:32821 was elected leader, sending a full tablet report...
I20260928 06:46:09.504556 18122 ts_tablet_manager.cc:1428] T 7ebac70a574647c3aaa0636caa0d50d9 P e22b2e87d56c406f9934d548ef018e33: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260928 06:46:09.505882 18124 raft_consensus.cc:348] T 7ebac70a574647c3aaa0636caa0d50d9 P 730a5444c89041b78ee2113d2ea2b47f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730a5444c89041b78ee2113d2ea2b47f" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 37379 } } peers { permanent_uuid: "e22b2e87d56c406f9934d548ef018e33" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33995 } } peers { permanent_uuid: "80ecadd1915b41cc86a82c7c56004884" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36543 } }
I20260928 06:46:09.506150 18124 raft_consensus.cc:374] T 7ebac70a574647c3aaa0636caa0d50d9 P 730a5444c89041b78ee2113d2ea2b47f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:46:09.506223 18124 raft_consensus.cc:729] T 7ebac70a574647c3aaa0636caa0d50d9 P 730a5444c89041b78ee2113d2ea2b47f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730a5444c89041b78ee2113d2ea2b47f, State: Initialized, Role: FOLLOWER
I20260928 06:46:09.506716 18124 consensus_queue.cc:260] T 7ebac70a574647c3aaa0636caa0d50d9 P 730a5444c89041b78ee2113d2ea2b47f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730a5444c89041b78ee2113d2ea2b47f" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 37379 } } peers { permanent_uuid: "e22b2e87d56c406f9934d548ef018e33" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33995 } } peers { permanent_uuid: "80ecadd1915b41cc86a82c7c56004884" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36543 } }
I20260928 06:46:09.507413 18104 heartbeater.cc:503] Master 127.7.48.62:32821 was elected leader, sending a full tablet report...
I20260928 06:46:09.507833 18124 ts_tablet_manager.cc:1428] T 7ebac70a574647c3aaa0636caa0d50d9 P 730a5444c89041b78ee2113d2ea2b47f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 06:46:09.635107 18127 raft_consensus.cc:482] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:46:09.635329 18127 raft_consensus.cc:504] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730a5444c89041b78ee2113d2ea2b47f" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 37379 } } peers { permanent_uuid: "e22b2e87d56c406f9934d548ef018e33" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33995 } } peers { permanent_uuid: "80ecadd1915b41cc86a82c7c56004884" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36543 } }
I20260928 06:46:09.636631 18127 leader_election.cc:288] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730a5444c89041b78ee2113d2ea2b47f (127.7.48.3:37379), e22b2e87d56c406f9934d548ef018e33 (127.7.48.2:33995)
I20260928 06:46:09.640905 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:09.636885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:37379 (local address 127.7.48.1:49635)
0928 06:46:09.637248 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:09.637263 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:09.637392 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:09.637740 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:09.637743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:09.637765 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:46:09.637901 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:09.637908 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.638707 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.638710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:09.639642 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:09.639651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.640449 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.640456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:09.640554 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:09.640578 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 06:46:09.640654 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:46:09.640709 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":77,"thread_start_us":155,"threads_started":1}
I20260928 06:46:09.640910 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:09.637013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:49635 (local address 127.7.48.3:37379)
0928 06:46:09.637290 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:09.637297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:09.637321 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:46:09.637568 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:09.637574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:09.637639 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:09.637698 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:46:09.638011 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.638587 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.639796 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.640329 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.640411 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:09.640495 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:09.640702 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 06:46:09.640785 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260928 06:46:09.640976 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:09.637035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:38961 (local address 127.7.48.2:33995)
0928 06:46:09.637197 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:09.637202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:09.637472 (+ 270us) server_negotiation.cc:407] Connection header received
0928 06:46:09.637632 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:09.637634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:09.637697 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:09.637777 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:46:09.638090 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.638604 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.639762 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.640398 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.640504 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:09.640569 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:09.640781 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 06:46:09.640849 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 06:46:09.640944 18131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:09.636895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:33995 (local address 127.7.48.1:38961)
0928 06:46:09.637371 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:09.637382 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:09.637504 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:09.637823 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:09.637825 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:09.637839 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:46:09.637936 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:09.637940 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.638742 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.638745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:09.639639 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:09.639648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.640530 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.640540 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:09.640621 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:09.640674 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 06:46:09.640749 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 06:46:09.640812 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":9,"thread_start_us":136,"threads_started":1}
I20260928 06:46:09.641830 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ebac70a574647c3aaa0636caa0d50d9" candidate_uuid: "80ecadd1915b41cc86a82c7c56004884" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730a5444c89041b78ee2113d2ea2b47f" is_pre_election: true
I20260928 06:46:09.641829 17924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ebac70a574647c3aaa0636caa0d50d9" candidate_uuid: "80ecadd1915b41cc86a82c7c56004884" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e22b2e87d56c406f9934d548ef018e33" is_pre_election: true
I20260928 06:46:09.642187 17924 raft_consensus.cc:2454] T 7ebac70a574647c3aaa0636caa0d50d9 P e22b2e87d56c406f9934d548ef018e33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80ecadd1915b41cc86a82c7c56004884 in term 0.
I20260928 06:46:09.642187 18058 raft_consensus.cc:2454] T 7ebac70a574647c3aaa0636caa0d50d9 P 730a5444c89041b78ee2113d2ea2b47f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80ecadd1915b41cc86a82c7c56004884 in term 0.
I20260928 06:46:09.642719 17724 leader_election.cc:302] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80ecadd1915b41cc86a82c7c56004884, e22b2e87d56c406f9934d548ef018e33; no voters:
I20260928 06:46:09.643064 18127 raft_consensus.cc:2781] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:46:09.643167 18127 raft_consensus.cc:482] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:46:09.643232 18127 raft_consensus.cc:3037] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:46:09.644374 18127 raft_consensus.cc:504] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730a5444c89041b78ee2113d2ea2b47f" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 37379 } } peers { permanent_uuid: "e22b2e87d56c406f9934d548ef018e33" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33995 } } peers { permanent_uuid: "80ecadd1915b41cc86a82c7c56004884" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36543 } }
I20260928 06:46:09.644761 18127 leader_election.cc:288] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884 [CANDIDATE]: Term 1 election: Requested vote from peers 730a5444c89041b78ee2113d2ea2b47f (127.7.48.3:37379), e22b2e87d56c406f9934d548ef018e33 (127.7.48.2:33995)
I20260928 06:46:09.645171 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ebac70a574647c3aaa0636caa0d50d9" candidate_uuid: "80ecadd1915b41cc86a82c7c56004884" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730a5444c89041b78ee2113d2ea2b47f"
I20260928 06:46:09.645272 17924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ebac70a574647c3aaa0636caa0d50d9" candidate_uuid: "80ecadd1915b41cc86a82c7c56004884" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e22b2e87d56c406f9934d548ef018e33"
I20260928 06:46:09.645316 18058 raft_consensus.cc:3037] T 7ebac70a574647c3aaa0636caa0d50d9 P 730a5444c89041b78ee2113d2ea2b47f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:46:09.645397 17924 raft_consensus.cc:3037] T 7ebac70a574647c3aaa0636caa0d50d9 P e22b2e87d56c406f9934d548ef018e33 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:46:09.646692 18058 raft_consensus.cc:2454] T 7ebac70a574647c3aaa0636caa0d50d9 P 730a5444c89041b78ee2113d2ea2b47f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80ecadd1915b41cc86a82c7c56004884 in term 1.
I20260928 06:46:09.646692 17924 raft_consensus.cc:2454] T 7ebac70a574647c3aaa0636caa0d50d9 P e22b2e87d56c406f9934d548ef018e33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80ecadd1915b41cc86a82c7c56004884 in term 1.
I20260928 06:46:09.646992 17724 leader_election.cc:302] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80ecadd1915b41cc86a82c7c56004884, e22b2e87d56c406f9934d548ef018e33; no voters:
I20260928 06:46:09.647177 18127 raft_consensus.cc:2781] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:46:09.647449 18127 raft_consensus.cc:686] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884 [term 1 LEADER]: Becoming Leader. State: Replica: 80ecadd1915b41cc86a82c7c56004884, State: Running, Role: LEADER
I20260928 06:46:09.647869 18127 consensus_queue.cc:237] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730a5444c89041b78ee2113d2ea2b47f" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 37379 } } peers { permanent_uuid: "e22b2e87d56c406f9934d548ef018e33" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33995 } } peers { permanent_uuid: "80ecadd1915b41cc86a82c7c56004884" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36543 } }
I20260928 06:46:09.651432 17645 catalog_manager.cc:5495] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80ecadd1915b41cc86a82c7c56004884 (127.7.48.1). New cstate: current_term: 1 leader_uuid: "80ecadd1915b41cc86a82c7c56004884" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730a5444c89041b78ee2113d2ea2b47f" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 37379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e22b2e87d56c406f9934d548ef018e33" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80ecadd1915b41cc86a82c7c56004884" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36543 } health_report { overall_health: HEALTHY } } }
I20260928 06:46:09.673142 18120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:09.669620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58006 (local address 127.7.48.1:36543)
0928 06:46:09.669750 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:09.669754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:09.669765 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:46:09.669845 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:09.669847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:09.669901 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:09.669997 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:46:09.670243 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.670732 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.671591 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:09.671792 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:09.671870 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:09.672219 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:46:09.672799 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:46:09.672923 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:09.672979 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 06:46:09.673027 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 06:46:09.675992 18105 tablet.cc:2369] T 7ebac70a574647c3aaa0636caa0d50d9 P 730a5444c89041b78ee2113d2ea2b47f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:46:09.679324 18058 raft_consensus.cc:1261] T 7ebac70a574647c3aaa0636caa0d50d9 P 730a5444c89041b78ee2113d2ea2b47f [term 1 FOLLOWER]: Refusing update from remote peer 80ecadd1915b41cc86a82c7c56004884: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:46:09.679327 17924 raft_consensus.cc:1261] T 7ebac70a574647c3aaa0636caa0d50d9 P e22b2e87d56c406f9934d548ef018e33 [term 1 FOLLOWER]: Refusing update from remote peer 80ecadd1915b41cc86a82c7c56004884: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:46:09.680084 18132 consensus_queue.cc:1035] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730a5444c89041b78ee2113d2ea2b47f" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 37379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:46:09.680281 18127 consensus_queue.cc:1035] T 7ebac70a574647c3aaa0636caa0d50d9 P 80ecadd1915b41cc86a82c7c56004884 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e22b2e87d56c406f9934d548ef018e33" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 06:46:09.683619 17971 tablet.cc:2369] T 7ebac70a574647c3aaa0636caa0d50d9 P e22b2e87d56c406f9934d548ef018e33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:46:09.688640 18136 mvcc.cc:205] Tried to move back new op lower bound from 7334207363798896640 to 7334207363680653312. Current Snapshot: MvccSnapshot[applied={T|T < 7334207363798896640}]
I20260928 06:46:09.697515 17645 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:46:09.697707 17645 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:46:09.700378 17645 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d31689ac2384b3d87933dcce2f0169b: Sending DeleteTablet for 3 replicas of tablet 7ebac70a574647c3aaa0636caa0d50d9
I20260928 06:46:09.701217 17770 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ebac70a574647c3aaa0636caa0d50d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:46:09 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:57998
I20260928 06:46:09.701333 17904 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ebac70a574647c3aaa0636caa0d50d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:46:09 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:50346
I20260928 06:46:09.702091 18038 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ebac70a574647c3aaa0636caa0d50d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:46:09 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:50478
I20260928 06:46:09.702610 18147 tablet_replica.cc:330] T 7ebac70a574647c3aaa0636caa0d50d9 P 730a5444c89041b78ee2113d2ea2b47f: stopping tablet replica
I20260928 06:46:09.702646 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 17707
I20260928 06:46:09.702967 18147 raft_consensus.cc:2229] T 7ebac70a574647c3aaa0636caa0d50d9 P 730a5444c89041b78ee2113d2ea2b47f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:46:09.703306 18147 raft_consensus.cc:2258] T 7ebac70a574647c3aaa0636caa0d50d9 P 730a5444c89041b78ee2113d2ea2b47f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:46:09.703429 18146 tablet_replica.cc:330] T 7ebac70a574647c3aaa0636caa0d50d9 P e22b2e87d56c406f9934d548ef018e33: stopping tablet replica
I20260928 06:46:09.703835 18146 raft_consensus.cc:2229] T 7ebac70a574647c3aaa0636caa0d50d9 P e22b2e87d56c406f9934d548ef018e33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:46:09.704154 18146 raft_consensus.cc:2258] T 7ebac70a574647c3aaa0636caa0d50d9 P e22b2e87d56c406f9934d548ef018e33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:46:09.705346 18147 ts_tablet_manager.cc:1905] T 7ebac70a574647c3aaa0636caa0d50d9 P 730a5444c89041b78ee2113d2ea2b47f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:46:09.706362 18146 ts_tablet_manager.cc:1905] T 7ebac70a574647c3aaa0636caa0d50d9 P e22b2e87d56c406f9934d548ef018e33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:46:09.708141 18147 ts_tablet_manager.cc:1918] T 7ebac70a574647c3aaa0636caa0d50d9 P 730a5444c89041b78ee2113d2ea2b47f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:46:09.708280 18147 log.cc:1196] T 7ebac70a574647c3aaa0636caa0d50d9 P 730a5444c89041b78ee2113d2ea2b47f: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/7ebac70a574647c3aaa0636caa0d50d9
I20260928 06:46:09.708660 18147 ts_tablet_manager.cc:1939] T 7ebac70a574647c3aaa0636caa0d50d9 P 730a5444c89041b78ee2113d2ea2b47f: Deleting consensus metadata
I20260928 06:46:09.709443 18146 ts_tablet_manager.cc:1918] T 7ebac70a574647c3aaa0636caa0d50d9 P e22b2e87d56c406f9934d548ef018e33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:46:09.709561 18146 log.cc:1196] T 7ebac70a574647c3aaa0636caa0d50d9 P e22b2e87d56c406f9934d548ef018e33: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/7ebac70a574647c3aaa0636caa0d50d9
I20260928 06:46:09.709923 18146 ts_tablet_manager.cc:1939] T 7ebac70a574647c3aaa0636caa0d50d9 P e22b2e87d56c406f9934d548ef018e33: Deleting consensus metadata
I20260928 06:46:09.710088 17629 catalog_manager.cc:4841] TS 730a5444c89041b78ee2113d2ea2b47f (127.7.48.3:37379): tablet 7ebac70a574647c3aaa0636caa0d50d9 (table test-table [id=688cd9b063a1412d99491a382371da94]) successfully deleted
I20260928 06:46:09.711072 17630 catalog_manager.cc:4841] TS e22b2e87d56c406f9934d548ef018e33 (127.7.48.2:33995): tablet 7ebac70a574647c3aaa0636caa0d50d9 (table test-table [id=688cd9b063a1412d99491a382371da94]) successfully deleted
W20260928 06:46:09.712020 17629 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.48.1:36543 (error 108)
I20260928 06:46:09.712239 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 17841
I20260928 06:46:09.713357 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:09.712995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:36543 (local address 127.0.0.1:58020)
0928 06:46:09.713171 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:09.713253 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.1:36543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 06:46:09.713557 17629 catalog_manager.cc:4584] TS 80ecadd1915b41cc86a82c7c56004884 (127.7.48.1:36543): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7ebac70a574647c3aaa0636caa0d50d9: Network error: Client connection negotiation failed: client connection to 127.7.48.1:36543: connect: Connection refused (error 111)
I20260928 06:46:09.718639 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 17975
I20260928 06:46:09.724325 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 17613
2026-09-28T06:46:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2366 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:46:09 dist-test-slave-twl6 krb5kdc[18151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:46:09 dist-test-slave-twl6 krb5kdc[18151](info): set up 2 sockets
Sep 28 06:46:09 dist-test-slave-twl6 krb5kdc[18151](info): commencing operation
krb5kdc: starting...
W20260928 06:46:10.761240 7360 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:46:10 dist-test-slave-twl6 krb5kdc[18151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:46:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:46:10Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.7.48.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.7.48.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.7.48.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/rangeradmin_127.7.48.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.7.48.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/rangeradmin_127.7.48.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.7.48.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.7.48.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.7.48.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/rangerlookup_127.7.48.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.7.48.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/rangerlookup_127.7.48.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.48.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.7.48.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.48.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/HTTP_127.7.48.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.48.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/HTTP_127.7.48.20@KRBTEST.COM.keytab.
I20260928 06:46:10.926206 7360 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskipM9PC/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/postgres -l logfile start
2026-09-28 06:46:13.376 UTC [18181] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 06:46:13.376 UTC [18181] LOG: listening on IPv4 address "127.7.48.20", port 59487
2026-09-28 06:46:13.379 UTC [18181] LOG: listening on Unix socket "/tmp/.s.PGSQL.59487"
2026-09-28 06:46:13.401 UTC [18184] LOG: database system was shut down at 2026-09-28 06:46:11 UTC
2026-09-28 06:46:13.404 UTC [18181] LOG: database system is ready to accept connections
I20260928 06:46:14.357141 7360 mini_postgres.cc:96] Postgres bound to 59487
2026-09-28 06:46:14.363 UTC [18192] FATAL: database "slave" does not exist
127.7.48.20:59487 - accepting connections
I20260928 06:46:14.364243 7360 mini_ranger.cc:161] Starting Ranger...
I20260928 06:46:14.379668 7360 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 06:46:14.838557 7360 mini_ranger.cc:87] Created ranger Postgres database
I20260928 06:46:14.838667 7360 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ranger-admin
2026-09-28 06:46:15,119 [I] DB FLAVOR :POSTGRES
2026-09-28 06:46:15,119 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:46:15,119 [I] Checking connection..
2026-09-28 06:46:15,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:59487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 06:46:15,344 [I] Checking connection passed.
2026-09-28 06:46:15,344 [I] --------- Verifying version history table ---------
2026-09-28 06:46:15,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:59487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:46:15,603 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:46:15,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:59487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:46:15,839 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:46:15,839 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 06:46:15,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:59487/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 06:46:16.072 UTC [18256] WARNING: there is no transaction in progress
2026-09-28 06:46:16,078 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 06:46:16,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:59487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:46:16,337 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 06:46:16,338 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:46:16,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:59487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 06:46:16,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:59487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 06:46:16,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:59487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-twl6.c.gcp-upstream.internal','N') ;"
2026-09-28 06:46:17,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:59487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:46:17,346 [I] Table x_portal_user does not exist in database ranger
2026-09-28 06:46:17,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:59487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:46:17,595 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 06:46:17,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:59487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:46:17,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:59487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:46:18,077 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 06:46:18,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:59487/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 06:46:18.974 UTC [18364] WARNING: there is no transaction in progress
2026-09-28 06:46:18.988 UTC [18364] WARNING: there is no transaction in progress
2026-09-28 06:46:19.003 UTC [18364] WARNING: there is no transaction in progress
2026-09-28 06:46:19.017 UTC [18364] WARNING: there is no transaction in progress
2026-09-28 06:46:19.034 UTC [18364] WARNING: there is no transaction in progress
2026-09-28 06:46:19.049 UTC [18364] WARNING: there is no transaction in progress
2026-09-28 06:46:19.097 UTC [18364] WARNING: there is no transaction in progress
2026-09-28 06:46:19.103 UTC [18364] WARNING: there is no transaction in progress
2026-09-28 06:46:19.116 UTC [18364] WARNING: there is no transaction in progress
2026-09-28 06:46:19.125 UTC [18364] WARNING: there is no transaction in progress
2026-09-28 06:46:19.134 UTC [18364] WARNING: there is no transaction in progress
2026-09-28 06:46:19.149 UTC [18364] WARNING: there is no transaction in progress
2026-09-28 06:46:19.157 UTC [18364] WARNING: there is no transaction in progress
2026-09-28 06:46:19.163 UTC [18364] WARNING: there is no transaction in progress
2026-09-28 06:46:19.169 UTC [18364] WARNING: there is no transaction in progress
2026-09-28 06:46:19.174 UTC [18364] WARNING: there is no transaction in progress
2026-09-28 06:46:19,787 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 06:46:19,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:59487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 06:46:20,006 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 06:46:20,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:59487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:46:20,251 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 06:46:20,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:59487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:46:20,508 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 06:46:20,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:59487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:46:20,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:59487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:46:20,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:59487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-twl6.c.gcp-upstream.internal';"
2026-09-28 06:46:21,209 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 06:46:21,209 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:59487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 06:46:21,458 [I] DB_PATCHES have already been applied
I20260928 06:46:21.464144 7360 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ranger-admin:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskipM9PC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 06:46:21.464214 7360 mini_ranger.cc:193] Using host: 127.7.48.20
I20260928 06:46:21.467624 7360 mini_ranger.cc:227] Ranger admin URL: http://127.7.48.20:45571
Sep 28, 2026 6:46:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:46:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:46:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 6:46:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 6:46:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 6:46:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 6:46:22 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 06:46:22.581025 7360 mini_ranger.cc:160] Time spent starting Ranger: real 8.217s user 0.000s sys 0.010s
Sep 28, 2026 6:46:42 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 6:46:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 6:46:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 6:46:42 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 6:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 6:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 6:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 6:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 6:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 6:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 6:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 6:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 6:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 6:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 6:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 6:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 6:46:42 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 6:46:43 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 06:46:43.736497 7360 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.7.48.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.7.48.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.7.48.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/rangerkms_127.7.48.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.7.48.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/rangerkms_127.7.48.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.48.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.7.48.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.48.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/HTTP_127.7.48.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.48.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/HTTP_127.7.48.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 06:46:43 dist-test-slave-twl6 krb5kdc[18151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578003, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 06:46:43.816390 7360 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 06:46:43.826103 7360 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 06:46:44.630401 7360 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 06:46:44.630482 7360 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ranger-kms
I20260928 06:46:44.630577 7360 mini_ranger_kms.cc:227] Using postgres at 127.7.48.20:59487
2026-09-28 06:46:44,905 [I] DB FLAVOR :POSTGRES
2026-09-28 06:46:44,905 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:46:44,905 [I] Checking connection
2026-09-28 06:46:44,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:59487/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 06:46:45,153 [I] connection success
2026-09-28 06:46:45,153 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 06:46:45,153 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 06:46:45,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:59487/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 06:46:45,379 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 06:46:45,379 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:46:45,379 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 06:46:45,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:59487/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 06:46:45,630 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 06:46:46.073483 7360 mini_ranger_kms.cc:310] Created kms service
I20260928 06:46:46.273938 7360 mini_ranger_kms.cc:326] Created kudu user
I20260928 06:46:46.319216 7360 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 06:46:46 dist-test-slave-twl6 krb5kdc[18151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578003, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.7.48.20@KRBTEST.COM
I20260928 06:46:46.716075 7360 mini_ranger_kms.cc:384] Added ranger policy
I20260928 06:46:46.716289 7360 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ranger-kms:/tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 06:46:46.716310 7360 mini_ranger_kms.cc:240] Using host: 127.7.48.20
I20260928 06:46:46.720044 7360 mini_ranger_kms.cc:276] Ranger KMS PID: 18544
I20260928 06:46:46.720180 7360 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.7.48.20:57551
Sep 28, 2026 6:46:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:46:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:46:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 6:46:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 6:46:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 6:46:48 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 06:46:48.980425 7360 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.164s user 0.002s sys 0.242s
I20260928 06:46:48.980564 7360 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 06:46:48.980679 7360 mini_ranger_kms.cc:401] 127.7.48.20:57551/kms/v1/keys
Sep 28 06:46:49 dist-test-slave-twl6 krb5kdc[18151](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790578009, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.48.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 06:46:49 dist-test-slave-twl6 krb5kdc[18151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578009, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.48.20@KRBTEST.COM for HTTP/127.7.48.20@KRBTEST.COM
Sep 28 06:46:49 dist-test-slave-twl6 krb5kdc[18151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578009, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.48.20@KRBTEST.COM for HTTP/127.7.48.20@KRBTEST.COM
Sep 28 06:46:51 dist-test-slave-twl6 krb5kdc[18151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:46:52.023559 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:46433
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:44401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:46433
--ranger_config_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.48.20:57551/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:46:52.135509 18595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:46:52.135802 18595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:46:52.135865 18595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:46:52.139645 18595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:46:52.139717 18595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:46:52.139752 18595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:46:52.139803 18595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:46:52.139838 18595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:46:52.145125 18595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:44401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/master-0/wal
--ranger_kms_url=127.7.48.20:57551/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:46433
--ranger_config_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:46433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:46:52.146174 18595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:46:52.147200 18595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:46:52.156117 18603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:52.156157 18601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:52.156239 18600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:46:52.156307 18595 server_base.cc:1034] running on GCE node
I20260928 06:46:52.156982 18595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:46:52.158069 18595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:46:52.159252 18595 hybrid_clock.cc:648] HybridClock initialized: now 1790578012159229 us; error 44 us; skew 500 ppm
Sep 28 06:46:52 dist-test-slave-twl6 krb5kdc[18151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578012, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:46:52.162994 18595 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:46:52.164157 18595 webserver.cc:460] Webserver started at http://127.7.48.62:45091/ using document root <none> and password file <none>
I20260928 06:46:52.164834 18595 fs_manager.cc:359] Metadata directory not provided
I20260928 06:46:52.164915 18595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:46:52.165162 18595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:46:52 dist-test-slave-twl6 krb5kdc[18151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578012, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for HTTP/127.7.48.20@KRBTEST.COM
I20260928 06:46:52.224830 18595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "3125d7b60c5a4073b8cf1782205744a1"
format_stamp: "Formatted at 2026-09-28 06:46:52 on dist-test-slave-twl6"
server_key: "d1123ca542e6f08054677833aebca7ef"
server_key_iv: "254f027f09048f3ecb3022be8cec08eb"
server_key_version: "kuduclusterkey@0"
I20260928 06:46:52.225575 18595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "3125d7b60c5a4073b8cf1782205744a1"
format_stamp: "Formatted at 2026-09-28 06:46:52 on dist-test-slave-twl6"
server_key: "d1123ca542e6f08054677833aebca7ef"
server_key_iv: "254f027f09048f3ecb3022be8cec08eb"
server_key_version: "kuduclusterkey@0"
I20260928 06:46:52.230561 18595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 06:46:52.246790 18610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:52.248297 18595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:46:52.248428 18595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "3125d7b60c5a4073b8cf1782205744a1"
format_stamp: "Formatted at 2026-09-28 06:46:52 on dist-test-slave-twl6"
server_key: "d1123ca542e6f08054677833aebca7ef"
server_key_iv: "254f027f09048f3ecb3022be8cec08eb"
server_key_version: "kuduclusterkey@0"
I20260928 06:46:52.248551 18595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:46:52.266728 18595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:46:52.269706 18595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:46:52.269894 18595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:46:52.278415 18595 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:46433
I20260928 06:46:52.278478 18662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:46433 every 8 connection(s)
I20260928 06:46:52.279393 18595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:46:52.281041 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 18595
I20260928 06:46:52.281181 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:46:52.281504 7360 external_mini_cluster.cc:1428] Setting key fb38168f68ccdaaa7e4d521984968dc5
I20260928 06:46:52.282341 18663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:46:52.289208 18663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3125d7b60c5a4073b8cf1782205744a1: Bootstrap starting.
I20260928 06:46:52.291563 18663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3125d7b60c5a4073b8cf1782205744a1: Neither blocks nor log segments found. Creating new log.
I20260928 06:46:52.292346 18663 log.cc:824] T 00000000000000000000000000000000 P 3125d7b60c5a4073b8cf1782205744a1: Log is configured to *not* fsync() on all Append() calls
Sep 28 06:46:52 dist-test-slave-twl6 krb5kdc[18151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:46:52.296371 18663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3125d7b60c5a4073b8cf1782205744a1: No bootstrap required, opened a new log
I20260928 06:46:52.298734 18666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:52.283056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56068 (local address 127.7.48.62:46433)
0928 06:46:52.283573 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:52.283598 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:52.283641 (+ 43us) server_negotiation.cc:407] Connection header received
0928 06:46:52.284542 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:52.284565 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:52.284976 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:52.285343 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:52.286206 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:52.287672 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:52.288909 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:52.289579 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:52.289734 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:52.293142 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:46:52.293147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:46:52.293160 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:46:52.293184 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:46:52.295919 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:46:52.296369 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:46:52.296375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:46:52.296381 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:46:52.296539 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:46:52.296814 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:46:52.296818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:46:52.296820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:46:52.297195 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:46:52.297386 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:52.297664 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 06:46:52.298049 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":194,"threads_started":1}
I20260928 06:46:52.300448 18663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3125d7b60c5a4073b8cf1782205744a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3125d7b60c5a4073b8cf1782205744a1" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 46433 } }
I20260928 06:46:52.300809 18663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3125d7b60c5a4073b8cf1782205744a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:46:52.300899 18663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3125d7b60c5a4073b8cf1782205744a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3125d7b60c5a4073b8cf1782205744a1, State: Initialized, Role: FOLLOWER
I20260928 06:46:52.301411 18663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3125d7b60c5a4073b8cf1782205744a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3125d7b60c5a4073b8cf1782205744a1" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 46433 } }
I20260928 06:46:52.301578 18663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3125d7b60c5a4073b8cf1782205744a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:46:52.301649 18663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3125d7b60c5a4073b8cf1782205744a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:46:52.301752 18663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3125d7b60c5a4073b8cf1782205744a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:46:52.302898 18663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3125d7b60c5a4073b8cf1782205744a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3125d7b60c5a4073b8cf1782205744a1" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 46433 } }
I20260928 06:46:52.303287 18663 leader_election.cc:302] T 00000000000000000000000000000000 P 3125d7b60c5a4073b8cf1782205744a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3125d7b60c5a4073b8cf1782205744a1; no voters:
I20260928 06:46:52.303675 18663 leader_election.cc:288] T 00000000000000000000000000000000 P 3125d7b60c5a4073b8cf1782205744a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:46:52.303763 18668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3125d7b60c5a4073b8cf1782205744a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:46:52.304060 18668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3125d7b60c5a4073b8cf1782205744a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3125d7b60c5a4073b8cf1782205744a1, State: Running, Role: LEADER
I20260928 06:46:52.304402 18668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3125d7b60c5a4073b8cf1782205744a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3125d7b60c5a4073b8cf1782205744a1" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 46433 } }
I20260928 06:46:52.305042 18663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3125d7b60c5a4073b8cf1782205744a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:46:52.305904 18669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3125d7b60c5a4073b8cf1782205744a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3125d7b60c5a4073b8cf1782205744a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3125d7b60c5a4073b8cf1782205744a1" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 46433 } } }
I20260928 06:46:52.306093 18669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3125d7b60c5a4073b8cf1782205744a1 [sys.catalog]: This master's current role is: LEADER
I20260928 06:46:52.306943 18669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3125d7b60c5a4073b8cf1782205744a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3125d7b60c5a4073b8cf1782205744a1. Latest consensus state: current_term: 1 leader_uuid: "3125d7b60c5a4073b8cf1782205744a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3125d7b60c5a4073b8cf1782205744a1" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 46433 } } }
I20260928 06:46:52.307032 18669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3125d7b60c5a4073b8cf1782205744a1 [sys.catalog]: This master's current role is: LEADER
I20260928 06:46:52.307278 18663 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 06:46:52.307760 18682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:46:52.313395 18682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:46:52.321231 18682 catalog_manager.cc:1347] Generated new cluster ID: 15cea79a958c4c2ab03f1259a2f10e64
I20260928 06:46:52.321303 18682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:46:52.342146 18682 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:46:52.343689 18682 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:46:52.349157 18682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3125d7b60c5a4073b8cf1782205744a1: Generated new TSK 0
I20260928 06:46:52.351083 18682 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
I20260928 06:46:53.113516 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.48.20:57551/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:46433
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:44401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:46:53.262781 18709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:46:53.263110 18709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:46:53.263212 18709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:46:53.267428 18709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:46:53.267547 18709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:46:53.267654 18709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:46:53.273520 18709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:44401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.7.48.20:57551/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:46433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:46:53.274646 18709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:46:53.275633 18709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:46:53.292053 18717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:53.292047 18716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:53.292300 18719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:46:53.292395 18709 server_base.cc:1034] running on GCE node
I20260928 06:46:53.292894 18709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:46:53.293682 18709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:46:53.294881 18709 hybrid_clock.cc:648] HybridClock initialized: now 1790578013294845 us; error 46 us; skew 500 ppm
Sep 28 06:46:53 dist-test-slave-twl6 krb5kdc[18151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578013, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:46:53.298141 18709 init.cc:377] Logged in from keytab as kudu/127.7.48.1@KRBTEST.COM (short username kudu)
I20260928 06:46:53.299098 18709 webserver.cc:460] Webserver started at http://127.7.48.1:44703/ using document root <none> and password file <none>
I20260928 06:46:53.299782 18709 fs_manager.cc:359] Metadata directory not provided
I20260928 06:46:53.299924 18709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:46:53.300199 18709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:46:53 dist-test-slave-twl6 krb5kdc[18151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578013, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for HTTP/127.7.48.20@KRBTEST.COM
I20260928 06:46:53.315068 18709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "c96908037dab4116ab9b1ee350bcbed7"
format_stamp: "Formatted at 2026-09-28 06:46:53 on dist-test-slave-twl6"
server_key: "e2df1515295bcb7dbcd44b10712fb197"
server_key_iv: "ab151c6a2e2d901f63eb19d70ae16212"
server_key_version: "kuduclusterkey@0"
I20260928 06:46:53.315848 18709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "c96908037dab4116ab9b1ee350bcbed7"
format_stamp: "Formatted at 2026-09-28 06:46:53 on dist-test-slave-twl6"
server_key: "e2df1515295bcb7dbcd44b10712fb197"
server_key_iv: "ab151c6a2e2d901f63eb19d70ae16212"
server_key_version: "kuduclusterkey@0"
I20260928 06:46:53.321327 18709 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 06:46:53.347237 18727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:53.348655 18709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:46:53.348800 18709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "c96908037dab4116ab9b1ee350bcbed7"
format_stamp: "Formatted at 2026-09-28 06:46:53 on dist-test-slave-twl6"
server_key: "e2df1515295bcb7dbcd44b10712fb197"
server_key_iv: "ab151c6a2e2d901f63eb19d70ae16212"
server_key_version: "kuduclusterkey@0"
I20260928 06:46:53.348940 18709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:46:53.373775 18709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:46:53.377838 18709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:46:53.378108 18709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:46:53.378916 18709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:46:53.380123 18709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:46:53.380228 18709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:53.380383 18709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:46:53.380463 18709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:53.392014 18709 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:36293
I20260928 06:46:53.392131 18841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:36293 every 8 connection(s)
I20260928 06:46:53.393278 18709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:46:53.397411 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 18709
I20260928 06:46:53.397529 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
I20260928 06:46:53.397800 7360 external_mini_cluster.cc:1428] Setting key c8f53f3f0371e15796fe613a5b059bbd
Sep 28 06:46:53 dist-test-slave-twl6 krb5kdc[18151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578013, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.48.2@KRBTEST.COM; defaulting to no policy
I20260928 06:46:53.411304 18845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:53.396240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:46433 (local address 127.7.48.1:35629)
0928 06:46:53.396785 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:53.396841 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:53.397890 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:53.398669 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:53.398678 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:53.399005 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:53.399602 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:53.399619 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:53.400794 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:53.400797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:53.402165 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:53.402175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:53.403253 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:53.403257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:53.403458 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:53.404129 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:46:53.404148 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:53.405734 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:46:53.407882 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:46:53.407888 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:46:53.407927 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:46:53.408292 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:46:53.408610 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:46:53.408613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:46:53.408615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:46:53.408802 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:46:53.409340 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:46:53.409347 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:46:53.409669 (+ 322us) client_negotiation.cc:769] Sending connection context
0928 06:46:53.409877 (+ 208us) client_negotiation.cc:240] Negotiation successful
0928 06:46:53.410220 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":148,"threads_started":1}
I20260928 06:46:53.411856 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:53.395696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:35629 (local address 127.7.48.62:46433)
0928 06:46:53.395954 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:53.395958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:53.396943 (+ 985us) server_negotiation.cc:407] Connection header received
0928 06:46:53.398136 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:53.398140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:53.398205 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:53.398286 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:53.399751 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:53.400636 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:53.402302 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:53.402650 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:53.402753 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:53.405914 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:46:53.405918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:46:53.405921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:46:53.405948 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:46:53.407749 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:46:53.408421 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:46:53.408424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:46:53.408426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:46:53.408509 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:46:53.408918 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:46:53.408921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:46:53.408924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:46:53.409180 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:46:53.409283 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:53.411307 (+ 2024us) server_negotiation.cc:299] Negotiation successful
0928 06:46:53.411653 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":58,"threads_started":1}
I20260928 06:46:53.413146 18842 heartbeater.cc:347] Connected to a master server at 127.7.48.62:46433
I20260928 06:46:53.413407 18842 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.2@KRBTEST.COM" created.
I20260928 06:46:53.414045 18842 heartbeater.cc:511] Master 127.7.48.62:46433 requested a full tablet report, sending...
I20260928 06:46:53.415824 18627 ts_manager.cc:194] Registered new tserver with Master: c96908037dab4116ab9b1ee350bcbed7 (127.7.48.1:36293)
I20260928 06:46:53.417315 18627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.1@KRBTEST.COM'} at 127.7.48.1:35629
Sep 28 06:46:53 dist-test-slave-twl6 krb5kdc[18151](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790578013, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
I20260928 06:46:53.431732 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.48.20:57551/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:46433
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:44401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:46:53.579912 18848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:46:53.580230 18848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:46:53.580305 18848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:46:53.586437 18848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:46:53.586517 18848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:46:53.586596 18848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:46:53.594123 18848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:44401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.7.48.20:57551/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:46433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:46:53.595432 18848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:46:53.596706 18848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:46:53.609355 18854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:53.609575 18855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:46:53.609912 18848 server_base.cc:1034] running on GCE node
W20260928 06:46:53.610031 18857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:46:53.610468 18848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:46:53.611212 18848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:46:53.612380 18848 hybrid_clock.cc:648] HybridClock initialized: now 1790578013612338 us; error 54 us; skew 500 ppm
Sep 28 06:46:53 dist-test-slave-twl6 krb5kdc[18151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578013, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:46:53.615661 18848 init.cc:377] Logged in from keytab as kudu/127.7.48.2@KRBTEST.COM (short username kudu)
I20260928 06:46:53.616758 18848 webserver.cc:460] Webserver started at http://127.7.48.2:43969/ using document root <none> and password file <none>
I20260928 06:46:53.617453 18848 fs_manager.cc:359] Metadata directory not provided
I20260928 06:46:53.617542 18848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:46:53.617767 18848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:46:53 dist-test-slave-twl6 krb5kdc[18151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578013, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for HTTP/127.7.48.20@KRBTEST.COM
I20260928 06:46:53.634310 18848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "298125a5819644e785bf133ab6ea9700"
format_stamp: "Formatted at 2026-09-28 06:46:53 on dist-test-slave-twl6"
server_key: "a6c3a058f19012b3d9b0596fc9dad36d"
server_key_iv: "89dca4501e266c4be38b1710524c805f"
server_key_version: "kuduclusterkey@0"
I20260928 06:46:53.635105 18848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "298125a5819644e785bf133ab6ea9700"
format_stamp: "Formatted at 2026-09-28 06:46:53 on dist-test-slave-twl6"
server_key: "a6c3a058f19012b3d9b0596fc9dad36d"
server_key_iv: "89dca4501e266c4be38b1710524c805f"
server_key_version: "kuduclusterkey@0"
I20260928 06:46:53.639959 18848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 06:46:53.655301 18864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:53.657160 18848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:46:53.657316 18848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "298125a5819644e785bf133ab6ea9700"
format_stamp: "Formatted at 2026-09-28 06:46:53 on dist-test-slave-twl6"
server_key: "a6c3a058f19012b3d9b0596fc9dad36d"
server_key_iv: "89dca4501e266c4be38b1710524c805f"
server_key_version: "kuduclusterkey@0"
I20260928 06:46:53.657500 18848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:46:53.695075 18848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:46:53.698791 18848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:46:53.698980 18848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:46:53.699743 18848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:46:53.700793 18848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:46:53.700858 18848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:53.700917 18848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:46:53.700943 18848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:53.714124 18848 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:33339
I20260928 06:46:53.715283 18848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:46:53.720654 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 18848
I20260928 06:46:53.720803 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:46:53.721132 7360 external_mini_cluster.cc:1428] Setting key 8ce98a72dbba3899f39a7345e3f0f947
I20260928 06:46:53.716869 18977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:33339 every 8 connection(s)
Sep 28 06:46:53 dist-test-slave-twl6 krb5kdc[18151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578013, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:46:53.746930 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:53.720197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:43963 (local address 127.7.48.62:46433)
0928 06:46:53.720359 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:53.720364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:53.728457 (+ 8093us) server_negotiation.cc:407] Connection header received
0928 06:46:53.733057 (+ 4600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:53.733062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:53.733124 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:53.733210 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:53.735245 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:53.736081 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:53.737912 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:53.738271 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:53.738344 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:53.742726 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:46:53.742730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:46:53.742733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:46:53.742765 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:46:53.744547 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:46:53.745179 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:46:53.745182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:46:53.745184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:46:53.745280 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:46:53.745648 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:46:53.745651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:46:53.745653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:46:53.745856 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:46:53.745923 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:53.746616 (+ 693us) server_negotiation.cc:299] Negotiation successful
0928 06:46:53.746751 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:46:53.748605 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:53.719090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:46433 (local address 127.7.48.2:43963)
0928 06:46:53.728277 (+ 9187us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:53.728343 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:53.732763 (+ 4420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:53.733643 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:53.733654 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:53.734058 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:53.735049 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:53.735072 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:53.736367 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:53.736370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:53.737771 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:53.737782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:53.739530 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:53.739533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:53.739741 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:53.740748 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:46:53.740770 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:53.742502 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:46:53.744692 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:46:53.744698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:46:53.744715 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:46:53.745058 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:46:53.745380 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:46:53.745383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:46:53.745385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:46:53.745545 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:46:53.746010 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:46:53.746016 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:46:53.746317 (+ 301us) client_negotiation.cc:769] Sending connection context
0928 06:46:53.747172 (+ 855us) client_negotiation.cc:240] Negotiation successful
0928 06:46:53.747500 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8910,"thread_start_us":150,"threads_started":1}
I20260928 06:46:53.750811 18978 heartbeater.cc:347] Connected to a master server at 127.7.48.62:46433
I20260928 06:46:53.751089 18978 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.7.48.3@KRBTEST.COM; defaulting to no policy
I20260928 06:46:53.751741 18978 heartbeater.cc:511] Master 127.7.48.62:46433 requested a full tablet report, sending...
I20260928 06:46:53.752889 18627 ts_manager.cc:194] Registered new tserver with Master: 298125a5819644e785bf133ab6ea9700 (127.7.48.2:33339)
I20260928 06:46:53.753706 18627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.2@KRBTEST.COM'} at 127.7.48.2:43963
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
I20260928 06:46:53.780205 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.48.20:57551/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:46433
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:44401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:46:53 dist-test-slave-twl6 krb5kdc[18151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578013, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for HTTP/127.7.48.20@KRBTEST.COM
W20260928 06:46:53.954147 18983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:46:53.954479 18983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:46:53.954537 18983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:46:53.960492 18983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:46:53.960572 18983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:46:53.960659 18983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:46:53.967561 18983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:44401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.7.48.20:57551/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:46433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:46:53.968935 18983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:46:53.970245 18983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:46:53.982707 18988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:53.983534 18991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:46:53.983803 18989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:46:53.983820 18983 server_base.cc:1034] running on GCE node
I20260928 06:46:53.984506 18983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:46:53.985318 18983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:46:53.992713 18983 hybrid_clock.cc:648] HybridClock initialized: now 1790578013992647 us; error 74 us; skew 500 ppm
Sep 28 06:46:53 dist-test-slave-twl6 krb5kdc[18151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578013, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:46:53.996292 18983 init.cc:377] Logged in from keytab as kudu/127.7.48.3@KRBTEST.COM (short username kudu)
I20260928 06:46:53.997380 18983 webserver.cc:460] Webserver started at http://127.7.48.3:36069/ using document root <none> and password file <none>
I20260928 06:46:53.998080 18983 fs_manager.cc:359] Metadata directory not provided
I20260928 06:46:53.998157 18983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:46:53.998373 18983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:46:53 dist-test-slave-twl6 krb5kdc[18151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578013, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for HTTP/127.7.48.20@KRBTEST.COM
I20260928 06:46:54.019423 18983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "1cf0b60c6d1a48578e898923c49b29c7"
format_stamp: "Formatted at 2026-09-28 06:46:54 on dist-test-slave-twl6"
server_key: "65d797d3aa639e703282af0988160c05"
server_key_iv: "2fdd4018ac8c23699dfdfb4dcda27212"
server_key_version: "kuduclusterkey@0"
I20260928 06:46:54.020218 18983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "1cf0b60c6d1a48578e898923c49b29c7"
format_stamp: "Formatted at 2026-09-28 06:46:54 on dist-test-slave-twl6"
server_key: "65d797d3aa639e703282af0988160c05"
server_key_iv: "2fdd4018ac8c23699dfdfb4dcda27212"
server_key_version: "kuduclusterkey@0"
I20260928 06:46:54.025429 18983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260928 06:46:54.052507 18998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:54.054198 18983 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260928 06:46:54.054343 18983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "1cf0b60c6d1a48578e898923c49b29c7"
format_stamp: "Formatted at 2026-09-28 06:46:54 on dist-test-slave-twl6"
server_key: "65d797d3aa639e703282af0988160c05"
server_key_iv: "2fdd4018ac8c23699dfdfb4dcda27212"
server_key_version: "kuduclusterkey@0"
I20260928 06:46:54.054466 18983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:46:54.084019 18983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:46:54.092429 18983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:46:54.092686 18983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:46:54.093448 18983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:46:54.094440 18983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:46:54.094554 18983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:54.094657 18983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:46:54.094732 18983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:46:54.107067 18983 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:36675
I20260928 06:46:54.108220 18983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:46:54.108866 19111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:36675 every 8 connection(s)
I20260928 06:46:54.116247 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 18983
I20260928 06:46:54.116369 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:46:54.116657 7360 external_mini_cluster.cc:1428] Setting key 4ffdbdf98049b45a18a88523a23c262f
Sep 28 06:46:54 dist-test-slave-twl6 krb5kdc[18151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578013, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:46:54.163484 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:54.112131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:55627 (local address 127.7.48.62:46433)
0928 06:46:54.112259 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:54.112263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:54.121696 (+ 9433us) server_negotiation.cc:407] Connection header received
0928 06:46:54.126264 (+ 4568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:54.126267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:54.126335 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:54.126380 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:54.135337 (+ 8957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.136129 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.144373 (+ 8244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.144704 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.144791 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:54.154489 (+ 9698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:46:54.154494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:46:54.154497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:46:54.154527 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:46:54.159866 (+ 5339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:46:54.161604 (+ 1738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:46:54.161607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:46:54.161609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:46:54.161700 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:46:54.162127 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:46:54.162130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:46:54.162132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:46:54.162366 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:46:54.162428 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:54.163172 (+ 744us) server_negotiation.cc:299] Negotiation successful
0928 06:46:54.163327 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 06:46:54.165297 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:54.111708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:46433 (local address 127.7.48.3:55627)
0928 06:46:54.121512 (+ 9804us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:54.121590 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:54.125499 (+ 3909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:54.134050 (+ 8551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:54.134062 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:54.134449 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:46:54.135174 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:54.135192 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.140425 (+ 5233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.140429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:54.141989 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:54.142001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.148109 (+ 6108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.148113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:54.148348 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:54.149752 (+ 1404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:46:54.149774 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:46:54.152863 (+ 3089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:46:54.160461 (+ 7598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:46:54.160469 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:46:54.160487 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:46:54.160880 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:46:54.161802 (+ 922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:46:54.161805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:46:54.161807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:46:54.162016 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:46:54.162551 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:46:54.162558 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:46:54.162904 (+ 346us) client_negotiation.cc:769] Sending connection context
0928 06:46:54.163939 (+ 1035us) client_negotiation.cc:240] Negotiation successful
0928 06:46:54.164290 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4131,"thread_start_us":3054,"threads_started":1}
Sep 28 06:46:54 dist-test-slave-twl6 krb5kdc[18151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578013, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for HTTP/127.7.48.20@KRBTEST.COM
I20260928 06:46:54.167250 19112 heartbeater.cc:347] Connected to a master server at 127.7.48.62:46433
I20260928 06:46:54.167483 19112 heartbeater.cc:464] Registering TS with master...
I20260928 06:46:54.168017 19112 heartbeater.cc:511] Master 127.7.48.62:46433 requested a full tablet report, sending...
I20260928 06:46:54.170302 18627 ts_manager.cc:194] Registered new tserver with Master: 1cf0b60c6d1a48578e898923c49b29c7 (127.7.48.3:36675)
I20260928 06:46:54.171033 18627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.3@KRBTEST.COM'} at 127.7.48.3:55627
I20260928 06:46:54.183380 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:46:54.203959 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:54.185671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56084 (local address 127.7.48.62:46433)
0928 06:46:54.185790 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:54.185794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:54.187589 (+ 1795us) server_negotiation.cc:407] Connection header received
0928 06:46:54.188125 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:54.188129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:54.188177 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:54.188226 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:46:54.190547 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.191322 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.196893 (+ 5571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.197230 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.197308 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:54.200298 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:46:54.200302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:46:54.200305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:46:54.200350 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:46:54.202122 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:46:54.202635 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:46:54.202640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:46:54.202642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:46:54.202730 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:46:54.203024 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:46:54.203028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:46:54.203030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:46:54.203214 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:46:54.203350 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:54.203633 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 06:46:54.203766 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:46:54.208200 18627 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:46:54.214726 18627 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:46:54.235560 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:54.225272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:36675 (local address 127.0.0.1:54128)
0928 06:46:54.227193 (+ 1921us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:54.227244 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:54.227365 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:54.228483 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:54.228488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:54.228509 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:46:54.228639 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:54.228645 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.231005 (+ 2360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.231008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:54.232786 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:54.232797 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.233985 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.233999 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:54.235123 (+ 1124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:54.235174 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 06:46:54.235262 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:46:54.235405 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1082,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260928 06:46:54.236676 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:54.226053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54128 (local address 127.7.48.3:36675)
0928 06:46:54.227668 (+ 1615us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:54.227680 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:54.227701 (+ 21us) server_negotiation.cc:407] Connection header received
0928 06:46:54.227778 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:54.227784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:54.228022 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:54.228165 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:46:54.228981 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.230091 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.232973 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.233831 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.236229 (+ 2398us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:54.236385 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:54.236437 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 06:46:54.236528 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1521,"thread_start_us":126,"threads_started":1}
I20260928 06:46:54.240350 19046 tablet_service.cc:1467] Processing CreateTablet for tablet ed11ade48cab4fd4af1f75e059aa1619 (DEFAULT_TABLE table=test-table [id=4fd13858a38d4d9ca4f28b82716fecb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:46:54.244652 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:54.225391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:33339 (local address 127.0.0.1:33400)
0928 06:46:54.227583 (+ 2192us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:54.227595 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:54.227672 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:54.233999 (+ 6327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:54.234001 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:54.234020 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:46:54.234125 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:54.234134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.239592 (+ 5458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.239596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:54.241418 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:54.241430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.244214 (+ 2784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.244218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:54.244326 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:54.244350 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 06:46:54.244436 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:46:54.244510 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2138,"thread_start_us":91,"threads_started":1}
I20260928 06:46:54.249430 19046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed11ade48cab4fd4af1f75e059aa1619. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:46:54.249701 19131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:54.226720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:36293 (local address 127.0.0.1:58612)
0928 06:46:54.227819 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:54.227830 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:54.227988 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:54.228572 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:54.228574 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:54.228586 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:46:54.228647 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:54.228655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.245693 (+ 17038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.245699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:54.247033 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:54.247044 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.248904 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.248908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:54.249382 (+ 474us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:54.249410 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 06:46:54.249486 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:46:54.249559 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1053,"mutex_wait_us":1,"thread_start_us":81,"threads_started":1}
I20260928 06:46:54.250463 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:54.226053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33400 (local address 127.7.48.2:33339)
0928 06:46:54.230377 (+ 4324us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:54.230394 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:54.230418 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:46:54.230488 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:54.230493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:54.230672 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:54.230817 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:46:54.234271 (+ 3454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.237757 (+ 3486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.243027 (+ 5270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.243848 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.243985 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:54.244149 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:54.250205 (+ 6056us) server_negotiation.cc:299] Negotiation successful
0928 06:46:54.250321 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4197,"thread_start_us":121,"threads_started":1}
I20260928 06:46:54.256189 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:54.227099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58612 (local address 127.7.48.1:36293)
0928 06:46:54.227490 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:54.227497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:54.227928 (+ 431us) server_negotiation.cc:407] Connection header received
0928 06:46:54.228126 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:54.228131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:54.228296 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:54.228428 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:46:54.228772 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.245526 (+ 16754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.247181 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.248601 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.248706 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:54.248843 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:54.255908 (+ 7065us) server_negotiation.cc:299] Negotiation successful
0928 06:46:54.256039 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":63,"threads_started":1}
I20260928 06:46:54.257671 18911 tablet_service.cc:1467] Processing CreateTablet for tablet ed11ade48cab4fd4af1f75e059aa1619 (DEFAULT_TABLE table=test-table [id=4fd13858a38d4d9ca4f28b82716fecb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:46:54.258765 18911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed11ade48cab4fd4af1f75e059aa1619. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:46:54.259222 18776 tablet_service.cc:1467] Processing CreateTablet for tablet ed11ade48cab4fd4af1f75e059aa1619 (DEFAULT_TABLE table=test-table [id=4fd13858a38d4d9ca4f28b82716fecb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:46:54.260239 18776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed11ade48cab4fd4af1f75e059aa1619. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:46:54.269387 19135 tablet_bootstrap.cc:492] T ed11ade48cab4fd4af1f75e059aa1619 P c96908037dab4116ab9b1ee350bcbed7: Bootstrap starting.
I20260928 06:46:54.273423 19135 tablet_bootstrap.cc:654] T ed11ade48cab4fd4af1f75e059aa1619 P c96908037dab4116ab9b1ee350bcbed7: Neither blocks nor log segments found. Creating new log.
I20260928 06:46:54.274470 19135 log.cc:824] T ed11ade48cab4fd4af1f75e059aa1619 P c96908037dab4116ab9b1ee350bcbed7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:46:54.276913 19135 tablet_bootstrap.cc:492] T ed11ade48cab4fd4af1f75e059aa1619 P c96908037dab4116ab9b1ee350bcbed7: No bootstrap required, opened a new log
I20260928 06:46:54.277347 19135 ts_tablet_manager.cc:1397] T ed11ade48cab4fd4af1f75e059aa1619 P c96908037dab4116ab9b1ee350bcbed7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 06:46:54.283473 19135 raft_consensus.cc:348] T ed11ade48cab4fd4af1f75e059aa1619 P c96908037dab4116ab9b1ee350bcbed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf0b60c6d1a48578e898923c49b29c7" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36675 } } peers { permanent_uuid: "298125a5819644e785bf133ab6ea9700" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33339 } } peers { permanent_uuid: "c96908037dab4116ab9b1ee350bcbed7" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36293 } }
I20260928 06:46:54.283972 19135 raft_consensus.cc:374] T ed11ade48cab4fd4af1f75e059aa1619 P c96908037dab4116ab9b1ee350bcbed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:46:54.284149 19135 raft_consensus.cc:729] T ed11ade48cab4fd4af1f75e059aa1619 P c96908037dab4116ab9b1ee350bcbed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c96908037dab4116ab9b1ee350bcbed7, State: Initialized, Role: FOLLOWER
I20260928 06:46:54.284854 19135 consensus_queue.cc:260] T ed11ade48cab4fd4af1f75e059aa1619 P c96908037dab4116ab9b1ee350bcbed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf0b60c6d1a48578e898923c49b29c7" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36675 } } peers { permanent_uuid: "298125a5819644e785bf133ab6ea9700" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33339 } } peers { permanent_uuid: "c96908037dab4116ab9b1ee350bcbed7" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36293 } }
I20260928 06:46:54.286183 19135 ts_tablet_manager.cc:1428] T ed11ade48cab4fd4af1f75e059aa1619 P c96908037dab4116ab9b1ee350bcbed7: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260928 06:46:54.286634 18842 heartbeater.cc:503] Master 127.7.48.62:46433 was elected leader, sending a full tablet report...
I20260928 06:46:54.288158 19134 tablet_bootstrap.cc:492] T ed11ade48cab4fd4af1f75e059aa1619 P 298125a5819644e785bf133ab6ea9700: Bootstrap starting.
I20260928 06:46:54.288633 19133 tablet_bootstrap.cc:492] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7: Bootstrap starting.
I20260928 06:46:54.291440 19133 tablet_bootstrap.cc:654] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7: Neither blocks nor log segments found. Creating new log.
I20260928 06:46:54.292418 19133 log.cc:824] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:46:54.294750 19133 tablet_bootstrap.cc:492] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7: No bootstrap required, opened a new log
I20260928 06:46:54.294970 19133 ts_tablet_manager.cc:1397] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 06:46:54.295820 19134 tablet_bootstrap.cc:654] T ed11ade48cab4fd4af1f75e059aa1619 P 298125a5819644e785bf133ab6ea9700: Neither blocks nor log segments found. Creating new log.
I20260928 06:46:54.298913 19133 raft_consensus.cc:348] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf0b60c6d1a48578e898923c49b29c7" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36675 } } peers { permanent_uuid: "298125a5819644e785bf133ab6ea9700" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33339 } } peers { permanent_uuid: "c96908037dab4116ab9b1ee350bcbed7" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36293 } }
I20260928 06:46:54.299226 19133 raft_consensus.cc:374] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:46:54.299296 19133 raft_consensus.cc:729] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf0b60c6d1a48578e898923c49b29c7, State: Initialized, Role: FOLLOWER
I20260928 06:46:54.300216 19134 log.cc:824] T ed11ade48cab4fd4af1f75e059aa1619 P 298125a5819644e785bf133ab6ea9700: Log is configured to *not* fsync() on all Append() calls
I20260928 06:46:54.302517 19133 consensus_queue.cc:260] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf0b60c6d1a48578e898923c49b29c7" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36675 } } peers { permanent_uuid: "298125a5819644e785bf133ab6ea9700" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33339 } } peers { permanent_uuid: "c96908037dab4116ab9b1ee350bcbed7" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36293 } }
I20260928 06:46:54.303586 19133 ts_tablet_manager.cc:1428] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260928 06:46:54.309414 19112 heartbeater.cc:503] Master 127.7.48.62:46433 was elected leader, sending a full tablet report...
I20260928 06:46:54.312064 19134 tablet_bootstrap.cc:492] T ed11ade48cab4fd4af1f75e059aa1619 P 298125a5819644e785bf133ab6ea9700: No bootstrap required, opened a new log
I20260928 06:46:54.312300 19134 ts_tablet_manager.cc:1397] T ed11ade48cab4fd4af1f75e059aa1619 P 298125a5819644e785bf133ab6ea9700: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.001s
I20260928 06:46:54.317045 19134 raft_consensus.cc:348] T ed11ade48cab4fd4af1f75e059aa1619 P 298125a5819644e785bf133ab6ea9700 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf0b60c6d1a48578e898923c49b29c7" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36675 } } peers { permanent_uuid: "298125a5819644e785bf133ab6ea9700" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33339 } } peers { permanent_uuid: "c96908037dab4116ab9b1ee350bcbed7" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36293 } }
I20260928 06:46:54.317423 19134 raft_consensus.cc:374] T ed11ade48cab4fd4af1f75e059aa1619 P 298125a5819644e785bf133ab6ea9700 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:46:54.317549 19134 raft_consensus.cc:729] T ed11ade48cab4fd4af1f75e059aa1619 P 298125a5819644e785bf133ab6ea9700 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 298125a5819644e785bf133ab6ea9700, State: Initialized, Role: FOLLOWER
I20260928 06:46:54.318111 19134 consensus_queue.cc:260] T ed11ade48cab4fd4af1f75e059aa1619 P 298125a5819644e785bf133ab6ea9700 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf0b60c6d1a48578e898923c49b29c7" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36675 } } peers { permanent_uuid: "298125a5819644e785bf133ab6ea9700" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33339 } } peers { permanent_uuid: "c96908037dab4116ab9b1ee350bcbed7" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36293 } }
I20260928 06:46:54.319389 19134 ts_tablet_manager.cc:1428] T ed11ade48cab4fd4af1f75e059aa1619 P 298125a5819644e785bf133ab6ea9700: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260928 06:46:54.319849 18978 heartbeater.cc:503] Master 127.7.48.62:46433 was elected leader, sending a full tablet report...
W20260928 06:46:54.359825 19113 tablet.cc:2369] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:46:54.396231 18843 tablet.cc:2369] T ed11ade48cab4fd4af1f75e059aa1619 P c96908037dab4116ab9b1ee350bcbed7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:46:54.471452 18979 tablet.cc:2369] T ed11ade48cab4fd4af1f75e059aa1619 P 298125a5819644e785bf133ab6ea9700: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:46:54.548309 19140 raft_consensus.cc:482] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:46:54.548563 19140 raft_consensus.cc:504] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf0b60c6d1a48578e898923c49b29c7" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36675 } } peers { permanent_uuid: "298125a5819644e785bf133ab6ea9700" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33339 } } peers { permanent_uuid: "c96908037dab4116ab9b1ee350bcbed7" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36293 } }
I20260928 06:46:54.549798 19140 leader_election.cc:288] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 298125a5819644e785bf133ab6ea9700 (127.7.48.2:33339), c96908037dab4116ab9b1ee350bcbed7 (127.7.48.1:36293)
I20260928 06:46:54.555428 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:54.550019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:33339 (local address 127.7.48.3:52391)
0928 06:46:54.550325 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:54.550343 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:54.550466 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:54.551319 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:54.551322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:54.551338 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:46:54.551441 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:54.551447 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.552461 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.552464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:54.553919 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:54.553925 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.554927 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.554930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:54.555127 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:54.555143 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 06:46:54.555219 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:46:54.555292 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":161}
I20260928 06:46:54.555428 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:54.550639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:33129 (local address 127.7.48.1:36293)
0928 06:46:54.550781 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:54.550785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:54.550799 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:46:54.550856 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:54.550861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:54.550914 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:54.551022 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:46:54.551438 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.552277 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.554046 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.554656 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.554747 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:54.554821 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:54.555264 (+ 443us) server_negotiation.cc:299] Negotiation successful
0928 06:46:54.555329 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 06:46:54.555907 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:54.550053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:36293 (local address 127.7.48.3:33129)
0928 06:46:54.550563 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:54.550577 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:46:54.550670 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:46:54.551160 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:46:54.551164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:46:54.551182 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:46:54.551307 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:54.551313 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.552461 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.552464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:54.553906 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:46:54.553917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.554936 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.554939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:46:54.555105 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:54.555144 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 06:46:54.555602 (+ 458us) client_negotiation.cc:240] Negotiation successful
0928 06:46:54.555687 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":87,"thread_start_us":127,"threads_started":1}
I20260928 06:46:54.556614 18796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed11ade48cab4fd4af1f75e059aa1619" candidate_uuid: "1cf0b60c6d1a48578e898923c49b29c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c96908037dab4116ab9b1ee350bcbed7" is_pre_election: true
I20260928 06:46:54.556721 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:54.550838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:52391 (local address 127.7.48.2:33339)
0928 06:46:54.550963 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:54.550970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:54.550984 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:46:54.551041 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:54.551044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:54.551089 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:54.551159 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:46:54.551555 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.552310 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.554046 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.554674 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.554764 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:54.554821 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:54.556542 (+ 1721us) server_negotiation.cc:299] Negotiation successful
0928 06:46:54.556617 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:46:54.556944 18796 raft_consensus.cc:2454] T ed11ade48cab4fd4af1f75e059aa1619 P c96908037dab4116ab9b1ee350bcbed7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cf0b60c6d1a48578e898923c49b29c7 in term 0.
I20260928 06:46:54.557303 18916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed11ade48cab4fd4af1f75e059aa1619" candidate_uuid: "1cf0b60c6d1a48578e898923c49b29c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "298125a5819644e785bf133ab6ea9700" is_pre_election: true
I20260928 06:46:54.557447 19002 leader_election.cc:302] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cf0b60c6d1a48578e898923c49b29c7, c96908037dab4116ab9b1ee350bcbed7; no voters:
I20260928 06:46:54.557585 18916 raft_consensus.cc:2454] T ed11ade48cab4fd4af1f75e059aa1619 P 298125a5819644e785bf133ab6ea9700 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cf0b60c6d1a48578e898923c49b29c7 in term 0.
I20260928 06:46:54.557758 19140 raft_consensus.cc:2781] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:46:54.557829 19140 raft_consensus.cc:482] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:46:54.557868 19140 raft_consensus.cc:3037] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:46:54.559139 19140 raft_consensus.cc:504] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf0b60c6d1a48578e898923c49b29c7" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36675 } } peers { permanent_uuid: "298125a5819644e785bf133ab6ea9700" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33339 } } peers { permanent_uuid: "c96908037dab4116ab9b1ee350bcbed7" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36293 } }
I20260928 06:46:54.559514 19140 leader_election.cc:288] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7 [CANDIDATE]: Term 1 election: Requested vote from peers 298125a5819644e785bf133ab6ea9700 (127.7.48.2:33339), c96908037dab4116ab9b1ee350bcbed7 (127.7.48.1:36293)
I20260928 06:46:54.559828 18916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed11ade48cab4fd4af1f75e059aa1619" candidate_uuid: "1cf0b60c6d1a48578e898923c49b29c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "298125a5819644e785bf133ab6ea9700"
I20260928 06:46:54.559932 18796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed11ade48cab4fd4af1f75e059aa1619" candidate_uuid: "1cf0b60c6d1a48578e898923c49b29c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c96908037dab4116ab9b1ee350bcbed7"
I20260928 06:46:54.559966 18916 raft_consensus.cc:3037] T ed11ade48cab4fd4af1f75e059aa1619 P 298125a5819644e785bf133ab6ea9700 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:46:54.560040 18796 raft_consensus.cc:3037] T ed11ade48cab4fd4af1f75e059aa1619 P c96908037dab4116ab9b1ee350bcbed7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:46:54.561245 18916 raft_consensus.cc:2454] T ed11ade48cab4fd4af1f75e059aa1619 P 298125a5819644e785bf133ab6ea9700 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cf0b60c6d1a48578e898923c49b29c7 in term 1.
I20260928 06:46:54.561286 18796 raft_consensus.cc:2454] T ed11ade48cab4fd4af1f75e059aa1619 P c96908037dab4116ab9b1ee350bcbed7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cf0b60c6d1a48578e898923c49b29c7 in term 1.
I20260928 06:46:54.561564 19000 leader_election.cc:302] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cf0b60c6d1a48578e898923c49b29c7, 298125a5819644e785bf133ab6ea9700; no voters:
I20260928 06:46:54.561796 19140 raft_consensus.cc:2781] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:46:54.562008 19140 raft_consensus.cc:686] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7 [term 1 LEADER]: Becoming Leader. State: Replica: 1cf0b60c6d1a48578e898923c49b29c7, State: Running, Role: LEADER
I20260928 06:46:54.562371 19140 consensus_queue.cc:237] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf0b60c6d1a48578e898923c49b29c7" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36675 } } peers { permanent_uuid: "298125a5819644e785bf133ab6ea9700" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33339 } } peers { permanent_uuid: "c96908037dab4116ab9b1ee350bcbed7" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36293 } }
I20260928 06:46:54.566040 18627 catalog_manager.cc:5495] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cf0b60c6d1a48578e898923c49b29c7 (127.7.48.3). New cstate: current_term: 1 leader_uuid: "1cf0b60c6d1a48578e898923c49b29c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf0b60c6d1a48578e898923c49b29c7" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 36675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "298125a5819644e785bf133ab6ea9700" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c96908037dab4116ab9b1ee350bcbed7" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36293 } health_report { overall_health: UNKNOWN } } }
I20260928 06:46:54.604175 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:54.599934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54664 (local address 127.7.48.3:36675)
0928 06:46:54.600059 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 06:46:54.600063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:46:54.600076 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:46:54.600170 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:46:54.600190 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:46:54.600250 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:46:54.600353 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:46:54.600654 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.601383 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.602495 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:46:54.602786 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:46:54.602867 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:46:54.603155 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:46:54.603724 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:46:54.603844 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 06:46:54.603959 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 06:46:54.604019 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:46:54.611361 18916 raft_consensus.cc:1261] T ed11ade48cab4fd4af1f75e059aa1619 P 298125a5819644e785bf133ab6ea9700 [term 1 FOLLOWER]: Refusing update from remote peer 1cf0b60c6d1a48578e898923c49b29c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:46:54.611472 18796 raft_consensus.cc:1261] T ed11ade48cab4fd4af1f75e059aa1619 P c96908037dab4116ab9b1ee350bcbed7 [term 1 FOLLOWER]: Refusing update from remote peer 1cf0b60c6d1a48578e898923c49b29c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:46:54.612254 19140 consensus_queue.cc:1035] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c96908037dab4116ab9b1ee350bcbed7" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 36293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:46:54.612427 19150 consensus_queue.cc:1035] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "298125a5819644e785bf133ab6ea9700" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:46:54.623025 19153 mvcc.cc:205] Tried to move back new op lower bound from 7334207547840098304 to 7334207547650707456. Current Snapshot: MvccSnapshot[applied={T|T < 7334207547840098304}]
I20260928 06:46:54.623276 19156 mvcc.cc:205] Tried to move back new op lower bound from 7334207547840098304 to 7334207547650707456. Current Snapshot: MvccSnapshot[applied={T|T < 7334207547840098304}]
I20260928 06:46:54.626638 19155 mvcc.cc:205] Tried to move back new op lower bound from 7334207547840098304 to 7334207547650707456. Current Snapshot: MvccSnapshot[applied={T|T < 7334207547840098304}]
I20260928 06:46:54.634526 18627 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:46:54.634713 18627 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:46:54.637034 18627 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3125d7b60c5a4073b8cf1782205744a1: Sending DeleteTablet for 3 replicas of tablet ed11ade48cab4fd4af1f75e059aa1619
I20260928 06:46:54.637884 18911 tablet_service.cc:1514] Processing DeleteTablet for tablet ed11ade48cab4fd4af1f75e059aa1619 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:46:54 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:33400
I20260928 06:46:54.638680 19046 tablet_service.cc:1514] Processing DeleteTablet for tablet ed11ade48cab4fd4af1f75e059aa1619 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:46:54 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:54128
I20260928 06:46:54.639596 18776 tablet_service.cc:1514] Processing DeleteTablet for tablet ed11ade48cab4fd4af1f75e059aa1619 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:46:54 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:58612
I20260928 06:46:54.640125 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 18709
I20260928 06:46:54.642545 19164 tablet_replica.cc:330] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7: stopping tablet replica
I20260928 06:46:54.642902 19164 raft_consensus.cc:2229] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:46:54.643172 19163 tablet_replica.cc:330] T ed11ade48cab4fd4af1f75e059aa1619 P 298125a5819644e785bf133ab6ea9700: stopping tablet replica
I20260928 06:46:54.643558 19164 raft_consensus.cc:2258] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:46:54.644084 19163 raft_consensus.cc:2229] T ed11ade48cab4fd4af1f75e059aa1619 P 298125a5819644e785bf133ab6ea9700 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:46:54.644441 19163 raft_consensus.cc:2258] T ed11ade48cab4fd4af1f75e059aa1619 P 298125a5819644e785bf133ab6ea9700 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:46:54.645896 19164 ts_tablet_manager.cc:1905] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:46:54.653560 19163 ts_tablet_manager.cc:1905] T ed11ade48cab4fd4af1f75e059aa1619 P 298125a5819644e785bf133ab6ea9700: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:46:54.655952 19164 ts_tablet_manager.cc:1918] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:46:54.658730 19163 ts_tablet_manager.cc:1918] T ed11ade48cab4fd4af1f75e059aa1619 P 298125a5819644e785bf133ab6ea9700: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:46:54.658895 19164 log.cc:1196] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/ed11ade48cab4fd4af1f75e059aa1619
I20260928 06:46:54.659315 19164 ts_tablet_manager.cc:1939] T ed11ade48cab4fd4af1f75e059aa1619 P 1cf0b60c6d1a48578e898923c49b29c7: Deleting consensus metadata
I20260928 06:46:54.659508 19163 log.cc:1196] T ed11ade48cab4fd4af1f75e059aa1619 P 298125a5819644e785bf133ab6ea9700: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/ed11ade48cab4fd4af1f75e059aa1619
I20260928 06:46:54.660123 19163 ts_tablet_manager.cc:1939] T ed11ade48cab4fd4af1f75e059aa1619 P 298125a5819644e785bf133ab6ea9700: Deleting consensus metadata
I20260928 06:46:54.662218 18612 catalog_manager.cc:4841] TS 298125a5819644e785bf133ab6ea9700 (127.7.48.2:33339): tablet ed11ade48cab4fd4af1f75e059aa1619 (table test-table [id=4fd13858a38d4d9ca4f28b82716fecb2]) successfully deleted
I20260928 06:46:54.663175 18611 catalog_manager.cc:4841] TS 1cf0b60c6d1a48578e898923c49b29c7 (127.7.48.3:36675): tablet ed11ade48cab4fd4af1f75e059aa1619 (table test-table [id=4fd13858a38d4d9ca4f28b82716fecb2]) successfully deleted
W20260928 06:46:54.663766 18614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.48.1:36293 (error 108)
I20260928 06:46:54.665207 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 18848
I20260928 06:46:54.665272 19131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:46:54.664968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:36293 (local address 127.0.0.1:42350)
0928 06:46:54.665106 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:46:54.665187 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.1:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 06:46:54.665863 18614 catalog_manager.cc:4584] TS c96908037dab4116ab9b1ee350bcbed7 (127.7.48.1:36293): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ed11ade48cab4fd4af1f75e059aa1619: Network error: Client connection negotiation failed: client connection to 127.7.48.1:36293: connect: Connection refused (error 111)
I20260928 06:46:54.673799 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 18983
I20260928 06:46:54.681496 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 18595
I20260928 06:46:54.688565 7360 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 06:46:55.033246 7360 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 06:46:55.033401 7360 mini_ranger.cc:66] Stopping Ranger...
I20260928 06:46:55.379171 7360 mini_ranger.cc:68] Stopped Ranger
2026-09-28 06:46:55.379 UTC [18181] LOG: received smart shutdown request
2026-09-28 06:46:55.382 UTC [18181] LOG: background worker "logical replication launcher" (PID 18190) exited with exit code 1
2026-09-28 06:46:55.383 UTC [18185] LOG: shutting down
2026-09-28 06:46:55.540 UTC [18181] LOG: database system is shut down
2026-09-28T06:46:55Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45928 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:46:55 dist-test-slave-twl6 krb5kdc[19175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:46:55 dist-test-slave-twl6 krb5kdc[19175](info): set up 2 sockets
Sep 28 06:46:55 dist-test-slave-twl6 krb5kdc[19175](info): commencing operation
krb5kdc: starting...
W20260928 06:46:56.672529 7360 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:46:56 dist-test-slave-twl6 krb5kdc[19175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:46:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:46:56Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.7.48.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.7.48.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.7.48.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/rangeradmin_127.7.48.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.7.48.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/rangeradmin_127.7.48.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.7.48.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.7.48.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.7.48.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/rangerlookup_127.7.48.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.7.48.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/rangerlookup_127.7.48.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.48.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.7.48.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.48.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/HTTP_127.7.48.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.48.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/HTTP_127.7.48.20@KRBTEST.COM.keytab.
I20260928 06:46:56.838052 7360 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskipM9PC/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/postgres -l logfile start
2026-09-28 06:46:59.155 UTC [19205] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 06:46:59.156 UTC [19205] LOG: listening on IPv4 address "127.7.48.20", port 49161
2026-09-28 06:46:59.159 UTC [19205] LOG: listening on Unix socket "/tmp/.s.PGSQL.49161"
2026-09-28 06:46:59.179 UTC [19208] LOG: database system was shut down at 2026-09-28 06:46:57 UTC
2026-09-28 06:46:59.183 UTC [19205] LOG: database system is ready to accept connections
I20260928 06:47:00.112221 7360 mini_postgres.cc:96] Postgres bound to 49161
2026-09-28 06:47:00.118 UTC [19216] FATAL: database "slave" does not exist
127.7.48.20:49161 - accepting connections
I20260928 06:47:00.119310 7360 mini_ranger.cc:161] Starting Ranger...
I20260928 06:47:00.134388 7360 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 06:47:00.603104 7360 mini_ranger.cc:87] Created ranger Postgres database
I20260928 06:47:00.603200 7360 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ranger-admin
2026-09-28 06:47:00,885 [I] DB FLAVOR :POSTGRES
2026-09-28 06:47:00,885 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:47:00,886 [I] Checking connection..
2026-09-28 06:47:00,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:49161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 06:47:01,138 [I] Checking connection passed.
2026-09-28 06:47:01,138 [I] --------- Verifying version history table ---------
2026-09-28 06:47:01,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:49161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:47:01,399 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:47:01,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:49161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:47:01,645 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:47:01,645 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 06:47:01,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:49161/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 06:47:01.900 UTC [19280] WARNING: there is no transaction in progress
2026-09-28 06:47:01,906 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 06:47:01,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:49161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:47:02,145 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 06:47:02,145 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:47:02,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:49161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 06:47:02,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:49161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 06:47:02,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:49161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-twl6.c.gcp-upstream.internal','N') ;"
2026-09-28 06:47:02,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:49161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:47:03,112 [I] Table x_portal_user does not exist in database ranger
2026-09-28 06:47:03,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:49161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:47:03,369 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 06:47:03,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:49161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:47:03,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:49161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:47:03,860 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 06:47:03,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:49161/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 06:47:04.821 UTC [19388] WARNING: there is no transaction in progress
2026-09-28 06:47:04.836 UTC [19388] WARNING: there is no transaction in progress
2026-09-28 06:47:04.853 UTC [19388] WARNING: there is no transaction in progress
2026-09-28 06:47:04.869 UTC [19388] WARNING: there is no transaction in progress
2026-09-28 06:47:04.887 UTC [19388] WARNING: there is no transaction in progress
2026-09-28 06:47:04.908 UTC [19388] WARNING: there is no transaction in progress
2026-09-28 06:47:04.972 UTC [19388] WARNING: there is no transaction in progress
2026-09-28 06:47:04.979 UTC [19388] WARNING: there is no transaction in progress
2026-09-28 06:47:04.992 UTC [19388] WARNING: there is no transaction in progress
2026-09-28 06:47:05.001 UTC [19388] WARNING: there is no transaction in progress
2026-09-28 06:47:05.011 UTC [19388] WARNING: there is no transaction in progress
2026-09-28 06:47:05.024 UTC [19388] WARNING: there is no transaction in progress
2026-09-28 06:47:05.033 UTC [19388] WARNING: there is no transaction in progress
2026-09-28 06:47:05.039 UTC [19388] WARNING: there is no transaction in progress
2026-09-28 06:47:05.046 UTC [19388] WARNING: there is no transaction in progress
2026-09-28 06:47:05.052 UTC [19388] WARNING: there is no transaction in progress
2026-09-28 06:47:05,634 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 06:47:05,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:49161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 06:47:05,873 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 06:47:05,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:49161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:47:06,128 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 06:47:06,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:49161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:47:06,352 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 06:47:06,353 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:49161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:47:06,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:49161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:47:06,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:49161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-twl6.c.gcp-upstream.internal';"
2026-09-28 06:47:07,074 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 06:47:07,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:49161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 06:47:07,312 [I] DB_PATCHES have already been applied
I20260928 06:47:07.318048 7360 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ranger-admin:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskipM9PC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 06:47:07.318137 7360 mini_ranger.cc:193] Using host: 127.7.48.20
I20260928 06:47:07.321545 7360 mini_ranger.cc:227] Ranger admin URL: http://127.7.48.20:43209
Sep 28, 2026 6:47:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:47:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:47:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 6:47:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 6:47:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 6:47:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 6:47:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 06:47:08.385583 7360 mini_ranger.cc:160] Time spent starting Ranger: real 8.266s user 0.003s sys 0.006s
Sep 28, 2026 6:47:26 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 6:47:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 28, 2026 6:47:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 6:47:27 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 6:47:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 6:47:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 6:47:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 6:47:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 6:47:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 6:47:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 6:47:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 6:47:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 6:47:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 6:47:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 6:47:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 6:47:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 6:47:27 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 6:47:27 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 06:47:28.331274 7360 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.7.48.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.7.48.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.7.48.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/rangerkms_127.7.48.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.7.48.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/rangerkms_127.7.48.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.48.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.7.48.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.48.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/HTTP_127.7.48.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.48.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/HTTP_127.7.48.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 06:47:28 dist-test-slave-twl6 krb5kdc[19175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578048, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 06:47:28.417541 7360 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 06:47:28.426787 7360 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 06:47:29.241662 7360 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 06:47:29.241750 7360 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ranger-kms
I20260928 06:47:29.241767 7360 mini_ranger_kms.cc:227] Using postgres at 127.7.48.20:49161
2026-09-28 06:47:29,414 [I] DB FLAVOR :POSTGRES
2026-09-28 06:47:29,414 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:47:29,414 [I] Checking connection
2026-09-28 06:47:29,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:49161/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 06:47:29,642 [I] connection success
2026-09-28 06:47:29,643 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 06:47:29,643 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 06:47:29,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:49161/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 06:47:29,880 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 06:47:29,880 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:47:29,880 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 06:47:29,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.48.20:49161/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 06:47:30,130 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 06:47:30.559463 7360 mini_ranger_kms.cc:310] Created kms service
I20260928 06:47:30.743470 7360 mini_ranger_kms.cc:326] Created kudu user
I20260928 06:47:30.799831 7360 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 06:47:30 dist-test-slave-twl6 krb5kdc[19175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578048, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.7.48.20@KRBTEST.COM
I20260928 06:47:31.221431 7360 mini_ranger_kms.cc:384] Added ranger policy
I20260928 06:47:31.221591 7360 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ranger-kms:/tmp/dist-test-taskipM9PC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskipM9PC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 06:47:31.221616 7360 mini_ranger_kms.cc:240] Using host: 127.7.48.20
I20260928 06:47:31.225991 7360 mini_ranger_kms.cc:276] Ranger KMS PID: 19568
I20260928 06:47:31.226082 7360 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.7.48.20:55785
Sep 28, 2026 6:47:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:47:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:47:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 6:47:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 6:47:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 6:47:32 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 06:47:33 dist-test-slave-twl6 krb5kdc[19175](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790578053, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.48.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:33.450753 7360 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.033s user 0.000s sys 0.158s
I20260928 06:47:33.450901 7360 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 06:47:33.451009 7360 mini_ranger_kms.cc:401] 127.7.48.20:55785/kms/v1/keys
Sep 28 06:47:33 dist-test-slave-twl6 krb5kdc[19175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578053, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.48.20@KRBTEST.COM for HTTP/127.7.48.20@KRBTEST.COM
Sep 28 06:47:34 dist-test-slave-twl6 krb5kdc[19175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578053, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.48.20@KRBTEST.COM for HTTP/127.7.48.20@KRBTEST.COM
Sep 28 06:47:36 dist-test-slave-twl6 krb5kdc[19175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:47:36.279249 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:45641
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:45641
--ranger_config_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.48.20:55785/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:47:36.385450 19619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:47:36.385733 19619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:47:36.385810 19619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:47:36.389586 19619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:47:36.389662 19619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:47:36.389696 19619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:47:36.389753 19619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:47:36.389786 19619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:47:36.395107 19619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/wal
--ranger_kms_url=127.7.48.20:55785/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:45641
--ranger_config_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:45641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:47:36.396152 19619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:47:36.397177 19619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:47:36.405725 19625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:36.405773 19624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:36.405773 19627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:36.405812 19619 server_base.cc:1034] running on GCE node
I20260928 06:47:36.406549 19619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:47:36.407647 19619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:47:36.408824 19619 hybrid_clock.cc:648] HybridClock initialized: now 1790578056408802 us; error 42 us; skew 500 ppm
Sep 28 06:47:36 dist-test-slave-twl6 krb5kdc[19175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578056, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:36.414853 19619 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:47:36.415727 19619 webserver.cc:460] Webserver started at http://127.7.48.62:42713/ using document root <none> and password file <none>
I20260928 06:47:36.416316 19619 fs_manager.cc:359] Metadata directory not provided
I20260928 06:47:36.416371 19619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:47:36.416579 19619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:47:36 dist-test-slave-twl6 krb5kdc[19175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578056, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for HTTP/127.7.48.20@KRBTEST.COM
I20260928 06:47:36.464691 19619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "c3c8888d139a4ff18f9431153b01dd0f"
format_stamp: "Formatted at 2026-09-28 06:47:36 on dist-test-slave-twl6"
server_key: "0e66735ca3522f269668083274ff0f64"
server_key_iv: "7ac0e85984a17562b86aaa39fa8ad37c"
server_key_version: "kuduclusterkey@0"
I20260928 06:47:36.465312 19619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "c3c8888d139a4ff18f9431153b01dd0f"
format_stamp: "Formatted at 2026-09-28 06:47:36 on dist-test-slave-twl6"
server_key: "0e66735ca3522f269668083274ff0f64"
server_key_iv: "7ac0e85984a17562b86aaa39fa8ad37c"
server_key_version: "kuduclusterkey@0"
I20260928 06:47:36.469808 19619 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:47:36.484074 19634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:36.485546 19619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:47:36.485700 19619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "c3c8888d139a4ff18f9431153b01dd0f"
format_stamp: "Formatted at 2026-09-28 06:47:36 on dist-test-slave-twl6"
server_key: "0e66735ca3522f269668083274ff0f64"
server_key_iv: "7ac0e85984a17562b86aaa39fa8ad37c"
server_key_version: "kuduclusterkey@0"
I20260928 06:47:36.485834 19619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:47:36.502653 19619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:47:36.505664 19619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:47:36.505847 19619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:47:36.513808 19619 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:45641
I20260928 06:47:36.513823 19686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:45641 every 8 connection(s)
I20260928 06:47:36.514777 19619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:47:36.516547 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 19619
I20260928 06:47:36.516664 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:47:36.517019 7360 external_mini_cluster.cc:1428] Setting key 244c59768978050cbc4222185ed5254e
I20260928 06:47:36.517916 19687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:47:36.524267 19687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f: Bootstrap starting.
I20260928 06:47:36.526993 19687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f: Neither blocks nor log segments found. Creating new log.
I20260928 06:47:36.527863 19687 log.cc:824] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f: Log is configured to *not* fsync() on all Append() calls
Sep 28 06:47:36 dist-test-slave-twl6 krb5kdc[19175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:47:36.530117 19687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f: No bootstrap required, opened a new log
I20260928 06:47:36.533399 19687 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c8888d139a4ff18f9431153b01dd0f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 45641 } }
I20260928 06:47:36.533691 19687 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:47:36.533802 19687 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3c8888d139a4ff18f9431153b01dd0f, State: Initialized, Role: FOLLOWER
I20260928 06:47:36.534353 19687 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c8888d139a4ff18f9431153b01dd0f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 45641 } }
I20260928 06:47:36.534497 19687 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:47:36.534584 19687 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:47:36.534680 19687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:47:36.535311 19690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:36.518725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55900 (local address 127.7.48.62:45641)
0928 06:47:36.521614 (+ 2889us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:36.521634 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:36.521674 (+ 40us) server_negotiation.cc:407] Connection header received
0928 06:47:36.522455 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:36.522479 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:36.522863 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:36.523204 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:36.524608 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:36.525882 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:36.526998 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:36.527537 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:36.527670 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:36.530640 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:36.530645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:36.530659 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:36.530683 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:36.532306 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:36.532767 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:36.532776 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:36.532781 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:36.532876 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:36.533164 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:36.533168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:36.533169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:36.533569 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:36.533820 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:36.534192 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 06:47:36.534503 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2757,"thread_start_us":114,"threads_started":1}
I20260928 06:47:36.535609 19687 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c8888d139a4ff18f9431153b01dd0f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 45641 } }
I20260928 06:47:36.536031 19687 leader_election.cc:302] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3c8888d139a4ff18f9431153b01dd0f; no voters:
I20260928 06:47:36.536350 19687 leader_election.cc:288] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:47:36.536417 19692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:47:36.536648 19692 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [term 1 LEADER]: Becoming Leader. State: Replica: c3c8888d139a4ff18f9431153b01dd0f, State: Running, Role: LEADER
I20260928 06:47:36.536989 19692 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c8888d139a4ff18f9431153b01dd0f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 45641 } }
I20260928 06:47:36.537607 19687 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:47:36.538162 19692 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3c8888d139a4ff18f9431153b01dd0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c8888d139a4ff18f9431153b01dd0f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 45641 } } }
I20260928 06:47:36.538352 19692 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [sys.catalog]: This master's current role is: LEADER
I20260928 06:47:36.538552 19693 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3c8888d139a4ff18f9431153b01dd0f. Latest consensus state: current_term: 1 leader_uuid: "c3c8888d139a4ff18f9431153b01dd0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c8888d139a4ff18f9431153b01dd0f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 45641 } } }
I20260928 06:47:36.538712 19693 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [sys.catalog]: This master's current role is: LEADER
I20260928 06:47:36.538784 19699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:47:36.541867 19699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:47:36.542282 19687 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 06:47:36.550675 19699 catalog_manager.cc:1347] Generated new cluster ID: 091f463ca777474bacdf63129335bd0c
I20260928 06:47:36.550747 19699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:47:36.566861 19699 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:47:36.567809 19699 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:47:36.573326 19699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f: Generated new TSK 0
I20260928 06:47:36.574258 19699 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
I20260928 06:47:37.240231 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.48.20:55785/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:45641
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:47:37.377403 19732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:47:37.377668 19732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:47:37.377712 19732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:47:37.383191 19732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:47:37.383275 19732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:47:37.383358 19732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:47:37.390934 19732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.7.48.20:55785/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:45641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:47:37.392233 19732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:47:37.393494 19732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:47:37.403471 19739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:37.403721 19740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:37.403956 19742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:37.408215 19732 server_base.cc:1034] running on GCE node
I20260928 06:47:37.408675 19732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:47:37.409374 19732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:47:37.410561 19732 hybrid_clock.cc:648] HybridClock initialized: now 1790578057410533 us; error 44 us; skew 500 ppm
Sep 28 06:47:37 dist-test-slave-twl6 krb5kdc[19175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578057, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:37.414462 19732 init.cc:377] Logged in from keytab as kudu/127.7.48.1@KRBTEST.COM (short username kudu)
I20260928 06:47:37.415477 19732 webserver.cc:460] Webserver started at http://127.7.48.1:33651/ using document root <none> and password file <none>
I20260928 06:47:37.416193 19732 fs_manager.cc:359] Metadata directory not provided
I20260928 06:47:37.416261 19732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:47:37.416486 19732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:47:37 dist-test-slave-twl6 krb5kdc[19175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578057, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for HTTP/127.7.48.20@KRBTEST.COM
I20260928 06:47:37.436338 19732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "88c89fa10457491187f58908b257489e"
format_stamp: "Formatted at 2026-09-28 06:47:37 on dist-test-slave-twl6"
server_key: "e4491991d379f0ee49fbe8c2a2c70612"
server_key_iv: "6beb8e829549d91521c0f7730fadab44"
server_key_version: "kuduclusterkey@0"
I20260928 06:47:37.438966 19732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "88c89fa10457491187f58908b257489e"
format_stamp: "Formatted at 2026-09-28 06:47:37 on dist-test-slave-twl6"
server_key: "e4491991d379f0ee49fbe8c2a2c70612"
server_key_iv: "6beb8e829549d91521c0f7730fadab44"
server_key_version: "kuduclusterkey@0"
I20260928 06:47:37.443986 19732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 06:47:37.468344 19751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:37.481416 19732 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.001s sys 0.000s
I20260928 06:47:37.481665 19732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "88c89fa10457491187f58908b257489e"
format_stamp: "Formatted at 2026-09-28 06:47:37 on dist-test-slave-twl6"
server_key: "e4491991d379f0ee49fbe8c2a2c70612"
server_key_iv: "6beb8e829549d91521c0f7730fadab44"
server_key_version: "kuduclusterkey@0"
I20260928 06:47:37.481930 19732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:47:37.503002 19732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:47:37.506805 19732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:47:37.507046 19732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:47:37.507782 19732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:47:37.509352 19732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:47:37.509455 19732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:37.509596 19732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:47:37.509675 19732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:47:37 dist-test-slave-twl6 krb5kdc[19175](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790578057, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:37.534821 19732 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:35889
I20260928 06:47:37.536149 19732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:47:37.539137 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 19732
I20260928 06:47:37.539264 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
I20260928 06:47:37.539597 7360 external_mini_cluster.cc:1428] Setting key ce6333bbf953dac463d1c2e888ed2c38
I20260928 06:47:37.546164 19864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:35889 every 8 connection(s)
WARNING: no policy specified for kudu/127.7.48.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.2@KRBTEST.COM" created.
Sep 28 06:47:37 dist-test-slave-twl6 krb5kdc[19175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578057, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:47:37.573207 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:37.548520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:45641 (local address 127.7.48.1:42059)
0928 06:47:37.556208 (+ 7688us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:37.556273 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:37.557323 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:37.558353 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:37.558362 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:37.558706 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:47:37.559399 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:37.559418 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:37.562523 (+ 3105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:37.562527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:37.563944 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:37.563955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:37.564818 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:37.564822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:37.565015 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:37.565734 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:47:37.565754 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:47:37.567523 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:47:37.569768 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:37.569774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:37.569791 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:37.570113 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:37.570434 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:37.570437 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:37.570439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:37.570627 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:37.571167 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:47:37.571174 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:47:37.571505 (+ 331us) client_negotiation.cc:769] Sending connection context
0928 06:47:37.571743 (+ 238us) client_negotiation.cc:240] Negotiation successful
0928 06:47:37.572113 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7417,"thread_start_us":175,"threads_started":1}
I20260928 06:47:37.573992 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:37.549654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:42059 (local address 127.7.48.62:45641)
0928 06:47:37.557802 (+ 8148us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:37.557810 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:37.557826 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:47:37.557870 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:37.557873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:37.557925 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:37.558018 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:37.559589 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:37.562354 (+ 2765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:37.564095 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:37.564509 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:37.564636 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:37.567718 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:37.567722 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:37.567725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:37.567751 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:37.569638 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:37.570226 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:37.570230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:37.570232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:37.570318 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:37.570765 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:37.570768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:37.570770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:37.571009 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:37.571108 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:37.573694 (+ 2586us) server_negotiation.cc:299] Negotiation successful
0928 06:47:37.573826 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8065,"thread_start_us":101,"threads_started":1}
I20260928 06:47:37.575135 19865 heartbeater.cc:347] Connected to a master server at 127.7.48.62:45641
I20260928 06:47:37.575418 19865 heartbeater.cc:464] Registering TS with master...
I20260928 06:47:37.576014 19865 heartbeater.cc:511] Master 127.7.48.62:45641 requested a full tablet report, sending...
I20260928 06:47:37.577703 19651 ts_manager.cc:194] Registered new tserver with Master: 88c89fa10457491187f58908b257489e (127.7.48.1:35889)
I20260928 06:47:37.579146 19651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.1@KRBTEST.COM'} at 127.7.48.1:42059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
I20260928 06:47:37.590859 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.48.20:55785/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:45641
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:47:37.749838 19871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:47:37.750321 19871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:47:37.750442 19871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:47:37.756850 19871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:47:37.757026 19871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:47:37.757189 19871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:47:37.765257 19871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.7.48.20:55785/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:45641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:47:37.766934 19871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:47:37.768414 19871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:47:37.781615 19878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:37.781844 19880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:37.781630 19877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:37.783061 19871 server_base.cc:1034] running on GCE node
I20260928 06:47:37.783520 19871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:47:37.784281 19871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:47:37.785461 19871 hybrid_clock.cc:648] HybridClock initialized: now 1790578057785432 us; error 53 us; skew 500 ppm
Sep 28 06:47:37 dist-test-slave-twl6 krb5kdc[19175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578057, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:37.788998 19871 init.cc:377] Logged in from keytab as kudu/127.7.48.2@KRBTEST.COM (short username kudu)
I20260928 06:47:37.790154 19871 webserver.cc:460] Webserver started at http://127.7.48.2:35857/ using document root <none> and password file <none>
I20260928 06:47:37.790904 19871 fs_manager.cc:359] Metadata directory not provided
I20260928 06:47:37.790989 19871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:47:37.791200 19871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:47:37 dist-test-slave-twl6 krb5kdc[19175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578057, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for HTTP/127.7.48.20@KRBTEST.COM
I20260928 06:47:37.805920 19871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "3cac415e9a4e4061bf487ff6ac3dae7a"
format_stamp: "Formatted at 2026-09-28 06:47:37 on dist-test-slave-twl6"
server_key: "abf3764a15696ba51fc13bee4630b833"
server_key_iv: "48a1cb0b6f0ad6d8df15125d035ce27a"
server_key_version: "kuduclusterkey@0"
I20260928 06:47:37.806558 19871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "3cac415e9a4e4061bf487ff6ac3dae7a"
format_stamp: "Formatted at 2026-09-28 06:47:37 on dist-test-slave-twl6"
server_key: "abf3764a15696ba51fc13bee4630b833"
server_key_iv: "48a1cb0b6f0ad6d8df15125d035ce27a"
server_key_version: "kuduclusterkey@0"
I20260928 06:47:37.811232 19871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 06:47:37.828194 19887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:37.829582 19871 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 06:47:37.829721 19871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "3cac415e9a4e4061bf487ff6ac3dae7a"
format_stamp: "Formatted at 2026-09-28 06:47:37 on dist-test-slave-twl6"
server_key: "abf3764a15696ba51fc13bee4630b833"
server_key_iv: "48a1cb0b6f0ad6d8df15125d035ce27a"
server_key_version: "kuduclusterkey@0"
I20260928 06:47:37.829869 19871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:47:37.871924 19871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:47:37.875613 19871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:47:37.875789 19871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:47:37.876525 19871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:47:37.877556 19871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:47:37.877605 19871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:37.877658 19871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:47:37.877679 19871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:37.891538 19871 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:33093
I20260928 06:47:37.892810 19871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:47:37.893671 20000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:33093 every 8 connection(s)
I20260928 06:47:37.902503 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 19871
I20260928 06:47:37.902627 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:47:37.902911 7360 external_mini_cluster.cc:1428] Setting key 81d95c603f43418f35eb11c46c1a9219
Sep 28 06:47:37 dist-test-slave-twl6 krb5kdc[19175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578057, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.48.3@KRBTEST.COM; defaulting to no policy
I20260928 06:47:37.919953 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:37.896272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:45641 (local address 127.7.48.2:42975)
0928 06:47:37.897548 (+ 1276us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:37.897603 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:37.898642 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:37.899428 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:37.899438 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:37.899801 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:47:37.900514 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:37.900535 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:37.901635 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:37.901638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:37.903180 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:37.903191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:37.903888 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:37.903891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:37.904138 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:37.912079 (+ 7941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:47:37.912105 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:47:37.913913 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:47:37.916390 (+ 2477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:37.916396 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:37.916414 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:37.916762 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:37.917137 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:37.917140 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:37.917142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:37.917329 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:37.917806 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:47:37.917831 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:47:37.918182 (+ 351us) client_negotiation.cc:769] Sending connection context
0928 06:47:37.918387 (+ 205us) client_negotiation.cc:240] Negotiation successful
0928 06:47:37.918720 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1074,"thread_start_us":139,"threads_started":1}
I20260928 06:47:37.920624 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:37.896114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:42975 (local address 127.7.48.62:45641)
0928 06:47:37.897283 (+ 1169us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:37.897288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:37.897701 (+ 413us) server_negotiation.cc:407] Connection header received
0928 06:47:37.898895 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:37.898899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:37.898965 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:37.899021 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:37.900703 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:37.901482 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:37.903309 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:37.903632 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:37.903706 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:37.914124 (+ 10418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:37.914128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:37.914131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:37.914163 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:37.916235 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:37.916901 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:37.916905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:37.916907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:37.917031 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:37.917433 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:37.917436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:37.917438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:37.917664 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:37.917754 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:37.920361 (+ 2607us) server_negotiation.cc:299] Negotiation successful
0928 06:47:37.920489 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1067}
I20260928 06:47:37.922349 20001 heartbeater.cc:347] Connected to a master server at 127.7.48.62:45641
I20260928 06:47:37.922624 20001 heartbeater.cc:464] Registering TS with master...
I20260928 06:47:37.923180 20001 heartbeater.cc:511] Master 127.7.48.62:45641 requested a full tablet report, sending...
I20260928 06:47:37.924329 19651 ts_manager.cc:194] Registered new tserver with Master: 3cac415e9a4e4061bf487ff6ac3dae7a (127.7.48.2:33093)
I20260928 06:47:37.925081 19651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.2@KRBTEST.COM'} at 127.7.48.2:42975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
I20260928 06:47:37.953289 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.48.20:55785/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:45641
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:47:38 dist-test-slave-twl6 krb5kdc[19175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578057, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for HTTP/127.7.48.20@KRBTEST.COM
W20260928 06:47:38.104651 20006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:47:38.104938 20006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:47:38.105016 20006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:47:38.109104 20006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:47:38.109189 20006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:47:38.109287 20006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:47:38.114662 20006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.7.48.20:55785/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:45641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:47:38.115965 20006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:47:38.117051 20006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:47:38.127485 20014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:38.127480 20011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:38.127982 20012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:38.128733 20006 server_base.cc:1034] running on GCE node
I20260928 06:47:38.129166 20006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:47:38.129873 20006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:47:38.131054 20006 hybrid_clock.cc:648] HybridClock initialized: now 1790578058131026 us; error 41 us; skew 500 ppm
Sep 28 06:47:38 dist-test-slave-twl6 krb5kdc[19175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578058, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:38.134603 20006 init.cc:377] Logged in from keytab as kudu/127.7.48.3@KRBTEST.COM (short username kudu)
I20260928 06:47:38.135699 20006 webserver.cc:460] Webserver started at http://127.7.48.3:44539/ using document root <none> and password file <none>
I20260928 06:47:38.136444 20006 fs_manager.cc:359] Metadata directory not provided
I20260928 06:47:38.136536 20006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:47:38.136767 20006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:47:38 dist-test-slave-twl6 krb5kdc[19175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578058, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for HTTP/127.7.48.20@KRBTEST.COM
I20260928 06:47:38.160188 20006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "fd094c845b1546e690f9369baa55e0c2"
format_stamp: "Formatted at 2026-09-28 06:47:38 on dist-test-slave-twl6"
server_key: "cc6c68c3fff154aba86249b1ab32d3df"
server_key_iv: "a48ac022ab6ca2a9c56f791bb750be0b"
server_key_version: "kuduclusterkey@0"
I20260928 06:47:38.161173 20006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "fd094c845b1546e690f9369baa55e0c2"
format_stamp: "Formatted at 2026-09-28 06:47:38 on dist-test-slave-twl6"
server_key: "cc6c68c3fff154aba86249b1ab32d3df"
server_key_iv: "a48ac022ab6ca2a9c56f791bb750be0b"
server_key_version: "kuduclusterkey@0"
I20260928 06:47:38.168340 20006 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.003s
I20260928 06:47:38.184963 20021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:38.186585 20006 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 06:47:38.186738 20006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "fd094c845b1546e690f9369baa55e0c2"
format_stamp: "Formatted at 2026-09-28 06:47:38 on dist-test-slave-twl6"
server_key: "cc6c68c3fff154aba86249b1ab32d3df"
server_key_iv: "a48ac022ab6ca2a9c56f791bb750be0b"
server_key_version: "kuduclusterkey@0"
I20260928 06:47:38.186878 20006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:47:38.207968 20006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:47:38.210904 20006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:47:38.211123 20006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:47:38.211782 20006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:47:38.212685 20006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:47:38.212755 20006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:38.212821 20006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:47:38.212875 20006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:38.223541 20006 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:46177
I20260928 06:47:38.223740 20134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:46177 every 8 connection(s)
I20260928 06:47:38.224571 20006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:47:38.226392 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 20006
I20260928 06:47:38.226497 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:47:38.226765 7360 external_mini_cluster.cc:1428] Setting key e64642e9d5db7e818248639b8118f9f5
Sep 28 06:47:38 dist-test-slave-twl6 krb5kdc[19175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578058, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:47:38.239621 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:38.227974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:54649 (local address 127.7.48.62:45641)
0928 06:47:38.228094 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:38.228100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:38.228981 (+ 881us) server_negotiation.cc:407] Connection header received
0928 06:47:38.229810 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:38.229813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:38.229860 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:38.229929 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:38.230953 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:38.231445 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:38.232659 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:38.232867 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:38.232970 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:38.235465 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:38.235468 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:38.235470 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:38.235496 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:38.236861 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:38.237378 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:38.237383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:38.237385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:38.237463 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:38.237785 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:38.237788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:38.237789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:38.238039 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:38.238146 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:38.238778 (+ 632us) server_negotiation.cc:299] Negotiation successful
0928 06:47:38.239448 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:47:38.240962 20137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:38.226892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:45641 (local address 127.7.48.3:54649)
0928 06:47:38.228851 (+ 1959us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:38.228889 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:38.229635 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:38.230113 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:38.230119 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:38.230368 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:47:38.230829 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:38.230840 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:38.231577 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:38.231581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:38.232507 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:38.232540 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:38.233010 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:38.233014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:38.233185 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:38.233888 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:47:38.233904 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:47:38.235277 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:47:38.236998 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:38.237003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:38.237017 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:38.237270 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:38.237551 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:38.237554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:38.237556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:38.237688 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:38.238143 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:47:38.238149 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:47:38.238473 (+ 324us) client_negotiation.cc:769] Sending connection context
0928 06:47:38.238699 (+ 226us) client_negotiation.cc:240] Negotiation successful
0928 06:47:38.239061 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1751,"thread_start_us":139,"threads_started":1}
I20260928 06:47:38.242149 20135 heartbeater.cc:347] Connected to a master server at 127.7.48.62:45641
I20260928 06:47:38.242415 20135 heartbeater.cc:464] Registering TS with master...
I20260928 06:47:38.242903 20135 heartbeater.cc:511] Master 127.7.48.62:45641 requested a full tablet report, sending...
I20260928 06:47:38.243968 19651 ts_manager.cc:194] Registered new tserver with Master: fd094c845b1546e690f9369baa55e0c2 (127.7.48.3:46177)
I20260928 06:47:38.244606 19651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.3@KRBTEST.COM'} at 127.7.48.3:54649
I20260928 06:47:38.251899 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:47:38.252686 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 19732
I20260928 06:47:38.259333 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 19871
I20260928 06:47:38.265147 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 20006
I20260928 06:47:38.270823 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 19619
I20260928 06:47:38.277441 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:45641
--webserver_interface=127.7.48.62
--webserver_port=42713
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:45641
--ranger_config_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.48.20:55785/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.48.20:55785/kms with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:47:38.381492 20138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:47:38.381742 20138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:47:38.381776 20138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:47:38.385679 20138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:47:38.385725 20138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:47:38.385744 20138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:47:38.385759 20138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:47:38.385773 20138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:47:38.390801 20138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.48.20:55785/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:45641
--ranger_config_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:45641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=42713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:47:38.391760 20138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:47:38.392773 20138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:47:38.401063 20146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:38.401063 20143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:38.401167 20138 server_base.cc:1034] running on GCE node
W20260928 06:47:38.401330 20144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:38.401861 20138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:47:38.402788 20138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:47:38.403945 20138 hybrid_clock.cc:648] HybridClock initialized: now 1790578058403916 us; error 39 us; skew 500 ppm
Sep 28 06:47:38 dist-test-slave-twl6 krb5kdc[19175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578058, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:38.406473 20138 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:47:38.407279 20138 webserver.cc:460] Webserver started at http://127.7.48.62:42713/ using document root <none> and password file <none>
I20260928 06:47:38.407800 20138 fs_manager.cc:359] Metadata directory not provided
I20260928 06:47:38.407882 20138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:47:38 dist-test-slave-twl6 krb5kdc[19175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578058, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for HTTP/127.7.48.20@KRBTEST.COM
I20260928 06:47:38.452010 20138 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:47:38.454145 20154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:38.455533 20138 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:47:38.455664 20138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "c3c8888d139a4ff18f9431153b01dd0f"
format_stamp: "Formatted at 2026-09-28 06:47:36 on dist-test-slave-twl6"
server_key: "0e66735ca3522f269668083274ff0f64"
server_key_iv: "7ac0e85984a17562b86aaa39fa8ad37c"
server_key_version: "kuduclusterkey@0"
I20260928 06:47:38.456272 20138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:47:38.478058 20138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:47:38.480513 20138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:47:38.480716 20138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:47:38.487746 20138 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:45641
I20260928 06:47:38.487752 20206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:45641 every 8 connection(s)
I20260928 06:47:38.488759 20138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:47:38.492762 20207 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:47:38.494314 20207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f: Bootstrap starting.
I20260928 06:47:38.495188 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 20138
I20260928 06:47:38.495625 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.48.20:55785/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:35889
--local_ip_for_outbound_sockets=127.7.48.1
--tserver_master_addrs=127.7.48.62:45641
--webserver_port=33651
--webserver_interface=127.7.48.1
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.48.20:55785/kms with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:47:38.510481 20207 log.cc:824] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:47:38.517032 20207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:47:38.517534 20207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f: Bootstrap complete.
I20260928 06:47:38.523854 20207 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c8888d139a4ff18f9431153b01dd0f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 45641 } }
I20260928 06:47:38.524377 20207 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3c8888d139a4ff18f9431153b01dd0f, State: Initialized, Role: FOLLOWER
I20260928 06:47:38.524887 20207 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c8888d139a4ff18f9431153b01dd0f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 45641 } }
I20260928 06:47:38.525038 20207 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:47:38.525105 20207 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:47:38.525197 20207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:47:38.526918 20207 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c8888d139a4ff18f9431153b01dd0f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 45641 } }
I20260928 06:47:38.527273 20207 leader_election.cc:302] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3c8888d139a4ff18f9431153b01dd0f; no voters:
I20260928 06:47:38.527590 20207 leader_election.cc:288] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:47:38.527667 20211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:47:38.527902 20211 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [term 2 LEADER]: Becoming Leader. State: Replica: c3c8888d139a4ff18f9431153b01dd0f, State: Running, Role: LEADER
I20260928 06:47:38.528172 20211 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c8888d139a4ff18f9431153b01dd0f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 45641 } }
I20260928 06:47:38.528805 20207 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:47:38.529098 20211 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3c8888d139a4ff18f9431153b01dd0f. Latest consensus state: current_term: 2 leader_uuid: "c3c8888d139a4ff18f9431153b01dd0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c8888d139a4ff18f9431153b01dd0f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 45641 } } }
I20260928 06:47:38.529225 20211 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [sys.catalog]: This master's current role is: LEADER
I20260928 06:47:38.529594 20212 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c3c8888d139a4ff18f9431153b01dd0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c8888d139a4ff18f9431153b01dd0f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 45641 } } }
I20260928 06:47:38.529786 20212 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f [sys.catalog]: This master's current role is: LEADER
I20260928 06:47:38.529647 20215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:47:38.533079 20215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:47:38.533601 20207 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 06:47:38.533831 20215 catalog_manager.cc:1259] Loaded cluster ID: 091f463ca777474bacdf63129335bd0c
I20260928 06:47:38.533922 20215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:47:38.536262 20215 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:47:38.536860 20215 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f: Loaded TSK: 0
I20260928 06:47:38.537992 20215 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:47:38.615320 20209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:47:38.615756 20209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:47:38.615847 20209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:47:38.620105 20209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:47:38.620185 20209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:47:38.620290 20209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:47:38.626055 20209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.48.20:55785/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:35889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=33651
--tserver_master_addrs=127.7.48.62:45641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:47:38.627178 20209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:47:38.628583 20209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:47:38.638902 20245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:38.639824 20247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:38.640185 20209 server_base.cc:1034] running on GCE node
W20260928 06:47:38.641021 20244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:38.641541 20209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:47:38.642280 20209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:47:38.643481 20209 hybrid_clock.cc:648] HybridClock initialized: now 1790578058643443 us; error 49 us; skew 500 ppm
Sep 28 06:47:38 dist-test-slave-twl6 krb5kdc[19175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578058, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:38.646618 20209 init.cc:377] Logged in from keytab as kudu/127.7.48.1@KRBTEST.COM (short username kudu)
I20260928 06:47:38.647612 20209 webserver.cc:460] Webserver started at http://127.7.48.1:33651/ using document root <none> and password file <none>
I20260928 06:47:38.648238 20209 fs_manager.cc:359] Metadata directory not provided
I20260928 06:47:38.648343 20209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:47:38 dist-test-slave-twl6 krb5kdc[19175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578058, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for HTTP/127.7.48.20@KRBTEST.COM
I20260928 06:47:38.668901 20209 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 06:47:38.672600 20255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:38.674113 20209 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 06:47:38.674250 20209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "88c89fa10457491187f58908b257489e"
format_stamp: "Formatted at 2026-09-28 06:47:37 on dist-test-slave-twl6"
server_key: "e4491991d379f0ee49fbe8c2a2c70612"
server_key_iv: "6beb8e829549d91521c0f7730fadab44"
server_key_version: "kuduclusterkey@0"
I20260928 06:47:38.674878 20209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:47:38.701375 20209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:47:38.704735 20209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:47:38.704982 20209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:47:38.705744 20209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:47:38.706730 20209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:47:38.706801 20209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:38.706867 20209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:47:38.706894 20209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:38.718245 20209 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:35889
I20260928 06:47:38.718456 20368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:35889 every 8 connection(s)
I20260928 06:47:38.719326 20209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:47:38.722357 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 20209
I20260928 06:47:38.722833 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.48.20:55785/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:33093
--local_ip_for_outbound_sockets=127.7.48.2
--tserver_master_addrs=127.7.48.62:45641
--webserver_port=35857
--webserver_interface=127.7.48.2
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.48.20:55785/kms with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:47:38 dist-test-slave-twl6 krb5kdc[19175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578058, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:47:38.745879 20372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:38.723208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:52669 (local address 127.7.48.62:45641)
0928 06:47:38.723786 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:38.723810 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:38.728142 (+ 4332us) server_negotiation.cc:407] Connection header received
0928 06:47:38.729583 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:38.729602 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:38.730023 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:38.730463 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:38.732384 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:38.733903 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:38.735666 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:38.736414 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:38.736565 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:38.739677 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:38.739685 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:38.739700 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:38.739729 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:38.741845 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:38.742484 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:38.742490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:38.742495 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:38.742635 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:38.743048 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:38.743052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:38.743055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:38.743545 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:38.743657 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:38.744478 (+ 821us) server_negotiation.cc:299] Negotiation successful
0928 06:47:38.744781 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":111,"threads_started":1}
I20260928 06:47:38.747918 20371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:38.721747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:45641 (local address 127.7.48.1:52669)
0928 06:47:38.724673 (+ 2926us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:38.724731 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:38.725986 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:38.731128 (+ 5142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:38.731150 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:38.731519 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:47:38.732190 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:38.732210 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:38.734075 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:38.734078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:38.735519 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:38.735531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:38.736824 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:38.736828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:38.737017 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:38.737711 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:47:38.737732 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:47:38.739466 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:47:38.741980 (+ 2514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:38.741985 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:38.741999 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:38.742344 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:38.742763 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:38.742766 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:38.742768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:38.742933 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:38.743742 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:47:38.743748 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:47:38.744090 (+ 342us) client_negotiation.cc:769] Sending connection context
0928 06:47:38.746661 (+ 2571us) client_negotiation.cc:240] Negotiation successful
0928 06:47:38.747023 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":152,"threads_started":1}
I20260928 06:47:38.751224 20369 heartbeater.cc:347] Connected to a master server at 127.7.48.62:45641
I20260928 06:47:38.751466 20369 heartbeater.cc:464] Registering TS with master...
I20260928 06:47:38.752105 20369 heartbeater.cc:511] Master 127.7.48.62:45641 requested a full tablet report, sending...
W20260928 06:47:38.760627 20369 heartbeater.cc:644] Failed to heartbeat to 127.7.48.62:45641 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 06:47:38.760758 20369 heartbeater.cc:464] Registering TS with master...
I20260928 06:47:38.760965 20369 heartbeater.cc:511] Master 127.7.48.62:45641 requested a full tablet report, sending...
I20260928 06:47:38.761729 20369 heartbeater.cc:464] Registering TS with master...
I20260928 06:47:38.761886 20369 heartbeater.cc:511] Master 127.7.48.62:45641 requested a full tablet report, sending...
W20260928 06:47:38.762666 20369 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:47:38.877795 20373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:47:38.878129 20373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:47:38.878196 20373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:47:38.883991 20373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:47:38.884068 20373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:47:38.884155 20373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:47:38.889443 20373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.48.20:55785/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:33093
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=35857
--tserver_master_addrs=127.7.48.62:45641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:47:38.890540 20373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:47:38.891724 20373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:47:38.902397 20379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:38.902447 20378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:38.902527 20381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:38.903226 20373 server_base.cc:1034] running on GCE node
I20260928 06:47:38.903638 20373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:47:38.904366 20373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:47:38.905541 20373 hybrid_clock.cc:648] HybridClock initialized: now 1790578058905516 us; error 49 us; skew 500 ppm
Sep 28 06:47:38 dist-test-slave-twl6 krb5kdc[19175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578058, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:38.908254 20373 init.cc:377] Logged in from keytab as kudu/127.7.48.2@KRBTEST.COM (short username kudu)
I20260928 06:47:38.909072 20373 webserver.cc:460] Webserver started at http://127.7.48.2:35857/ using document root <none> and password file <none>
I20260928 06:47:38.909600 20373 fs_manager.cc:359] Metadata directory not provided
I20260928 06:47:38.909711 20373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:47:38 dist-test-slave-twl6 krb5kdc[19175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578058, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for HTTP/127.7.48.20@KRBTEST.COM
I20260928 06:47:38.931097 20373 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:47:38.934088 20389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:38.935740 20373 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 06:47:38.935972 20373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "3cac415e9a4e4061bf487ff6ac3dae7a"
format_stamp: "Formatted at 2026-09-28 06:47:37 on dist-test-slave-twl6"
server_key: "abf3764a15696ba51fc13bee4630b833"
server_key_iv: "48a1cb0b6f0ad6d8df15125d035ce27a"
server_key_version: "kuduclusterkey@0"
I20260928 06:47:38.936697 20373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:47:38.952591 20373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:47:38.956432 20373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:47:38.956650 20373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:47:38.957386 20373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:47:38.958495 20373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:47:38.958554 20373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:38.958606 20373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:47:38.958626 20373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:38.969986 20373 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:33093
I20260928 06:47:38.970046 20502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:33093 every 8 connection(s)
I20260928 06:47:38.971237 20373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:47:38.976092 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 20373
I20260928 06:47:38.976559 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.48.20:55785/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:46177
--local_ip_for_outbound_sockets=127.7.48.3
--tserver_master_addrs=127.7.48.62:45641
--webserver_port=44539
--webserver_interface=127.7.48.3
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.48.20:55785/kms with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:47:38 dist-test-slave-twl6 krb5kdc[19175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578058, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:47:38.993125 20372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:38.973274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:38671 (local address 127.7.48.62:45641)
0928 06:47:38.973408 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:38.973413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:38.974624 (+ 1211us) server_negotiation.cc:407] Connection header received
0928 06:47:38.975960 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:38.975964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:38.976044 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:38.976115 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:38.977940 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:38.978676 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:38.980454 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:38.980722 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:38.980837 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:38.988757 (+ 7920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:38.988761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:38.988764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:38.988796 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:38.990684 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:38.991300 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:38.991303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:38.991305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:38.991391 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:38.991791 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:38.991794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:38.991796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:38.992065 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:38.992165 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:38.992800 (+ 635us) server_negotiation.cc:299] Negotiation successful
0928 06:47:38.992947 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:47:38.994098 20505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:38.973826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:45641 (local address 127.7.48.2:38671)
0928 06:47:38.974423 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:38.974486 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:38.975654 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:38.976588 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:38.976597 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:38.977164 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:47:38.977776 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:38.977794 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:38.978831 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:38.978834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:38.980283 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:38.980294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:38.980870 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:38.980874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:38.981050 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:38.984056 (+ 3006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:47:38.984083 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:47:38.988544 (+ 4461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:47:38.990818 (+ 2274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:38.990824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:38.990839 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:38.991173 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:38.991502 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:38.991505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:38.991507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:38.991668 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:38.992184 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:47:38.992191 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:47:38.992514 (+ 323us) client_negotiation.cc:769] Sending connection context
0928 06:47:38.992728 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 06:47:38.993061 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":152,"threads_started":1}
I20260928 06:47:38.995435 20503 heartbeater.cc:347] Connected to a master server at 127.7.48.62:45641
I20260928 06:47:38.995679 20503 heartbeater.cc:464] Registering TS with master...
I20260928 06:47:38.996264 20503 heartbeater.cc:511] Master 127.7.48.62:45641 requested a full tablet report, sending...
W20260928 06:47:38.997767 20503 heartbeater.cc:644] Failed to heartbeat to 127.7.48.62:45641 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 06:47:38.997881 20503 heartbeater.cc:464] Registering TS with master...
I20260928 06:47:38.998052 20503 heartbeater.cc:511] Master 127.7.48.62:45641 requested a full tablet report, sending...
I20260928 06:47:38.998782 20503 heartbeater.cc:464] Registering TS with master...
I20260928 06:47:38.998929 20503 heartbeater.cc:511] Master 127.7.48.62:45641 requested a full tablet report, sending...
W20260928 06:47:38.999598 20503 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:47:39.127823 20506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:47:39.128182 20506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:47:39.128257 20506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:47:39.132329 20506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:47:39.132408 20506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:47:39.132494 20506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:47:39.138016 20506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.48.20:55785/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:46177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=44539
--tserver_master_addrs=127.7.48.62:45641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:47:39.139071 20506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:47:39.140296 20506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:47:39.152042 20515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:39.152042 20513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:39.152221 20506 server_base.cc:1034] running on GCE node
W20260928 06:47:39.152838 20512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:39.153335 20506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:47:39.154112 20506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:47:39.155315 20506 hybrid_clock.cc:648] HybridClock initialized: now 1790578059155279 us; error 47 us; skew 500 ppm
Sep 28 06:47:39 dist-test-slave-twl6 krb5kdc[19175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578059, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:39.158121 20506 init.cc:377] Logged in from keytab as kudu/127.7.48.3@KRBTEST.COM (short username kudu)
I20260928 06:47:39.159039 20506 webserver.cc:460] Webserver started at http://127.7.48.3:44539/ using document root <none> and password file <none>
I20260928 06:47:39.159642 20506 fs_manager.cc:359] Metadata directory not provided
I20260928 06:47:39.159742 20506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:47:39 dist-test-slave-twl6 krb5kdc[19175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578059, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for HTTP/127.7.48.20@KRBTEST.COM
I20260928 06:47:39.186731 20506 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 06:47:39.189998 20523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:39.191627 20506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 06:47:39.191758 20506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "fd094c845b1546e690f9369baa55e0c2"
format_stamp: "Formatted at 2026-09-28 06:47:38 on dist-test-slave-twl6"
server_key: "cc6c68c3fff154aba86249b1ab32d3df"
server_key_iv: "a48ac022ab6ca2a9c56f791bb750be0b"
server_key_version: "kuduclusterkey@0"
I20260928 06:47:39.192500 20506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:47:39.211478 20506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:47:39.214983 20506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:47:39.215190 20506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:47:39.215875 20506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:47:39.216928 20506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:47:39.216997 20506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:39.217057 20506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:47:39.217088 20506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:39.227521 20506 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:46177
I20260928 06:47:39.231498 20636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:46177 every 8 connection(s)
I20260928 06:47:39.232474 20506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:47:39.240252 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 20506
Sep 28 06:47:39 dist-test-slave-twl6 krb5kdc[19175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578059, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:47:39.259996 20372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:39.235114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:39493 (local address 127.7.48.62:45641)
0928 06:47:39.235245 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:39.235250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:39.244473 (+ 9223us) server_negotiation.cc:407] Connection header received
0928 06:47:39.247583 (+ 3110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:39.247586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:39.247647 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:39.247733 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:39.249423 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:39.250240 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:39.251937 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:39.252234 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:39.252306 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:39.255544 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:39.255548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:39.255551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:39.255579 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:39.257455 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:39.258080 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:39.258084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:39.258086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:39.258171 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:39.258563 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:39.258566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:39.258568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:39.258812 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:39.258892 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:39.259667 (+ 775us) server_negotiation.cc:299] Negotiation successful
0928 06:47:39.259801 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:47:39.261951 20639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:39.235668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:45641 (local address 127.7.48.3:39493)
0928 06:47:39.244260 (+ 8592us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:39.244350 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:39.245487 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:39.248097 (+ 2610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:39.248109 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:39.248518 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:47:39.249233 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:39.249255 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:39.250407 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:39.250411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:39.251777 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:39.251789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:39.252523 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:39.252528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:39.252726 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:39.253463 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:47:39.253521 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:47:39.255316 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:47:39.257607 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:39.257613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:39.257631 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:39.257960 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:39.258277 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:39.258280 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:39.258282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:39.258443 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:39.258991 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:47:39.258997 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:47:39.259322 (+ 325us) client_negotiation.cc:769] Sending connection context
0928 06:47:39.260485 (+ 1163us) client_negotiation.cc:240] Negotiation successful
0928 06:47:39.260851 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8331,"thread_start_us":146,"threads_started":1}
I20260928 06:47:39.263751 20637 heartbeater.cc:347] Connected to a master server at 127.7.48.62:45641
I20260928 06:47:39.264052 20637 heartbeater.cc:464] Registering TS with master...
I20260928 06:47:39.264638 20637 heartbeater.cc:511] Master 127.7.48.62:45641 requested a full tablet report, sending...
W20260928 06:47:39.266319 20637 heartbeater.cc:644] Failed to heartbeat to 127.7.48.62:45641 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 06:47:39.266453 20637 heartbeater.cc:464] Registering TS with master...
I20260928 06:47:39.266654 20637 heartbeater.cc:511] Master 127.7.48.62:45641 requested a full tablet report, sending...
I20260928 06:47:39.267678 20637 heartbeater.cc:464] Registering TS with master...
I20260928 06:47:39.267848 20637 heartbeater.cc:511] Master 127.7.48.62:45641 requested a full tablet report, sending...
W20260928 06:47:39.268653 20637 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:47:39.269179 20641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:39.240954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55908 (local address 127.7.48.62:45641)
0928 06:47:39.247756 (+ 6802us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:39.247767 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:39.260044 (+ 12277us) server_negotiation.cc:407] Connection header received
0928 06:47:39.260088 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:39.260090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:39.260149 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:39.260289 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:39.261169 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:39.261982 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:39.263069 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:39.263413 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:39.263512 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:39.265235 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:39.265238 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:39.265241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:39.265264 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:39.267073 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:39.267586 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:39.267588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:39.267589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:39.267662 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:39.268039 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:39.268043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:39.268045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:39.268282 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:39.268483 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:39.268862 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 06:47:39.268977 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6719,"thread_start_us":96,"threads_started":1}
Sep 28 06:47:39 dist-test-slave-twl6 krb5kdc[19175](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790578059, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:39.762934 20369 heartbeater.cc:464] Registering TS with master...
I20260928 06:47:39.763262 20369 heartbeater.cc:511] Master 127.7.48.62:45641 requested a full tablet report, sending...
I20260928 06:47:39.764554 20171 ts_manager.cc:194] Registered new tserver with Master: 88c89fa10457491187f58908b257489e (127.7.48.1:35889)
I20260928 06:47:39.766216 20171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.1@KRBTEST.COM'} at 127.7.48.1:52669
Sep 28 06:47:39 dist-test-slave-twl6 krb5kdc[19175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578059, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for HTTP/127.7.48.20@KRBTEST.COM
I20260928 06:47:39.999801 20503 heartbeater.cc:464] Registering TS with master...
I20260928 06:47:40.000120 20503 heartbeater.cc:511] Master 127.7.48.62:45641 requested a full tablet report, sending...
I20260928 06:47:40.000849 20171 ts_manager.cc:194] Registered new tserver with Master: 3cac415e9a4e4061bf487ff6ac3dae7a (127.7.48.2:33093)
I20260928 06:47:40.001425 20171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.2@KRBTEST.COM'} at 127.7.48.2:38671
Sep 28 06:47:40 dist-test-slave-twl6 krb5kdc[19175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578059, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for HTTP/127.7.48.20@KRBTEST.COM
I20260928 06:47:40.269011 20637 heartbeater.cc:464] Registering TS with master...
I20260928 06:47:40.269344 20637 heartbeater.cc:511] Master 127.7.48.62:45641 requested a full tablet report, sending...
I20260928 06:47:40.270323 20171 ts_manager.cc:194] Registered new tserver with Master: fd094c845b1546e690f9369baa55e0c2 (127.7.48.3:46177)
I20260928 06:47:40.271070 20171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.3@KRBTEST.COM'} at 127.7.48.3:39493
I20260928 06:47:40.283987 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:47:40.312820 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:40.286660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55916 (local address 127.7.48.62:45641)
0928 06:47:40.287733 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:40.287739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:40.288168 (+ 429us) server_negotiation.cc:407] Connection header received
0928 06:47:40.288216 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:40.288219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:40.288270 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:40.288501 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:40.289558 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:40.290406 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:40.291557 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:40.291937 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:40.292043 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:40.308192 (+ 16149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:40.308197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:40.308200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:40.308245 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:40.310275 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:40.310847 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:40.310851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:40.310853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:40.310952 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:40.311348 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:40.311351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:40.311353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:40.311594 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:40.311723 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:40.312499 (+ 776us) server_negotiation.cc:299] Negotiation successful
0928 06:47:40.312633 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":987,"thread_start_us":90,"threads_started":1}
I20260928 06:47:40.317552 20171 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55916:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:47:40.322554 20171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:47:40.367269 20671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:40.351123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:46177 (local address 127.0.0.1:43780)
0928 06:47:40.356449 (+ 5326us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:40.356536 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:40.356669 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:40.359231 (+ 2562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:40.359238 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:40.359266 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:47:40.359421 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:40.359430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:40.361146 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:40.361149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:40.364673 (+ 3524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:40.364684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:40.366859 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:40.366862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:40.366946 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:40.366967 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 06:47:40.367052 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 06:47:40.367117 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5131,"mutex_wait_us":1,"thread_start_us":118,"threads_started":1}
I20260928 06:47:40.367708 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:40.351707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:35889 (local address 127.0.0.1:33944)
0928 06:47:40.356887 (+ 5180us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:40.356899 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:40.356976 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:40.359398 (+ 2422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:40.359402 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:40.359421 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:47:40.359558 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:40.359565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:40.362628 (+ 3063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:40.362632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:40.363813 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:40.363824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:40.366027 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:40.366030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:40.366145 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:40.366184 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 06:47:40.366292 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 06:47:40.366410 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5121,"thread_start_us":81,"threads_started":1}
I20260928 06:47:40.367918 20668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:40.351253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43780 (local address 127.7.48.3:46177)
0928 06:47:40.358705 (+ 7452us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:40.358711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:40.358736 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:47:40.358811 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:40.358816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:40.359038 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:40.359193 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:47:40.359697 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:40.360982 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:40.365780 (+ 4798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:40.366570 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:40.366672 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:40.366799 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:40.367646 (+ 847us) server_negotiation.cc:299] Negotiation successful
0928 06:47:40.367754 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6553,"thread_start_us":90,"threads_started":1}
I20260928 06:47:40.368490 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:40.352138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33944 (local address 127.7.48.1:35889)
0928 06:47:40.356167 (+ 4029us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:40.356175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:40.357071 (+ 896us) server_negotiation.cc:407] Connection header received
0928 06:47:40.357160 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:40.357166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:40.357421 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:40.357588 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:47:40.359696 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:40.361013 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:40.364876 (+ 3863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:40.365713 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:40.365818 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:40.365955 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:40.367346 (+ 1391us) server_negotiation.cc:299] Negotiation successful
0928 06:47:40.368373 (+ 1027us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3898,"thread_start_us":114,"threads_started":1}
I20260928 06:47:40.371162 20571 tablet_service.cc:1467] Processing CreateTablet for tablet bab56985d291408caa7778c85def24fa (DEFAULT_TABLE table=test-table [id=c5aaf6838d154528a16b4ef105843e2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:47:40.371397 20303 tablet_service.cc:1467] Processing CreateTablet for tablet bab56985d291408caa7778c85def24fa (DEFAULT_TABLE table=test-table [id=c5aaf6838d154528a16b4ef105843e2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:47:40.372319 20571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab56985d291408caa7778c85def24fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:47:40.372429 20303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab56985d291408caa7778c85def24fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:47:40.382158 20667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:40.353095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:33093 (local address 127.0.0.1:35976)
0928 06:47:40.361192 (+ 8097us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:40.361205 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:40.361283 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:40.361961 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:40.361963 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:40.361979 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:47:40.362057 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:40.362062 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:40.363349 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:40.363352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:40.364475 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:40.364486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:40.381680 (+ 17194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:40.381684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:40.381805 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:40.381836 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 06:47:40.381920 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 06:47:40.382006 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8042,"thread_start_us":87,"threads_started":1}
I20260928 06:47:40.382715 20672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:40.353511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35976 (local address 127.7.48.2:33093)
0928 06:47:40.358443 (+ 4932us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:40.358450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:40.361377 (+ 2927us) server_negotiation.cc:407] Connection header received
0928 06:47:40.361457 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:40.361462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:40.361658 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:40.361795 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:47:40.362180 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:40.363219 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:40.364623 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:40.365423 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:40.365556 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:40.365720 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:40.382428 (+ 16708us) server_negotiation.cc:299] Negotiation successful
0928 06:47:40.382576 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4814,"thread_start_us":80,"threads_started":1}
I20260928 06:47:40.383376 20673 tablet_bootstrap.cc:492] T bab56985d291408caa7778c85def24fa P fd094c845b1546e690f9369baa55e0c2: Bootstrap starting.
I20260928 06:47:40.385560 20437 tablet_service.cc:1467] Processing CreateTablet for tablet bab56985d291408caa7778c85def24fa (DEFAULT_TABLE table=test-table [id=c5aaf6838d154528a16b4ef105843e2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:47:40.386063 20673 tablet_bootstrap.cc:654] T bab56985d291408caa7778c85def24fa P fd094c845b1546e690f9369baa55e0c2: Neither blocks nor log segments found. Creating new log.
I20260928 06:47:40.386595 20437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab56985d291408caa7778c85def24fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:47:40.391053 20673 log.cc:824] T bab56985d291408caa7778c85def24fa P fd094c845b1546e690f9369baa55e0c2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:47:40.391782 20674 tablet_bootstrap.cc:492] T bab56985d291408caa7778c85def24fa P 88c89fa10457491187f58908b257489e: Bootstrap starting.
I20260928 06:47:40.395385 20676 tablet_bootstrap.cc:492] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a: Bootstrap starting.
I20260928 06:47:40.397537 20676 tablet_bootstrap.cc:654] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a: Neither blocks nor log segments found. Creating new log.
I20260928 06:47:40.398432 20676 log.cc:824] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:47:40.400502 20676 tablet_bootstrap.cc:492] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a: No bootstrap required, opened a new log
I20260928 06:47:40.400596 20674 tablet_bootstrap.cc:654] T bab56985d291408caa7778c85def24fa P 88c89fa10457491187f58908b257489e: Neither blocks nor log segments found. Creating new log.
I20260928 06:47:40.400687 20676 ts_tablet_manager.cc:1397] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 06:47:40.402052 20673 tablet_bootstrap.cc:492] T bab56985d291408caa7778c85def24fa P fd094c845b1546e690f9369baa55e0c2: No bootstrap required, opened a new log
I20260928 06:47:40.402263 20673 ts_tablet_manager.cc:1397] T bab56985d291408caa7778c85def24fa P fd094c845b1546e690f9369baa55e0c2: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20260928 06:47:40.404610 20676 raft_consensus.cc:348] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd094c845b1546e690f9369baa55e0c2" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 46177 } } peers { permanent_uuid: "88c89fa10457491187f58908b257489e" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35889 } } peers { permanent_uuid: "3cac415e9a4e4061bf487ff6ac3dae7a" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33093 } }
I20260928 06:47:40.404875 20676 raft_consensus.cc:374] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:47:40.404925 20676 raft_consensus.cc:729] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cac415e9a4e4061bf487ff6ac3dae7a, State: Initialized, Role: FOLLOWER
I20260928 06:47:40.405393 20676 consensus_queue.cc:260] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd094c845b1546e690f9369baa55e0c2" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 46177 } } peers { permanent_uuid: "88c89fa10457491187f58908b257489e" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35889 } } peers { permanent_uuid: "3cac415e9a4e4061bf487ff6ac3dae7a" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33093 } }
I20260928 06:47:40.406394 20676 ts_tablet_manager.cc:1428] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 06:47:40.406507 20673 raft_consensus.cc:348] T bab56985d291408caa7778c85def24fa P fd094c845b1546e690f9369baa55e0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd094c845b1546e690f9369baa55e0c2" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 46177 } } peers { permanent_uuid: "88c89fa10457491187f58908b257489e" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35889 } } peers { permanent_uuid: "3cac415e9a4e4061bf487ff6ac3dae7a" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33093 } }
I20260928 06:47:40.406752 20503 heartbeater.cc:503] Master 127.7.48.62:45641 was elected leader, sending a full tablet report...
I20260928 06:47:40.406759 20673 raft_consensus.cc:374] T bab56985d291408caa7778c85def24fa P fd094c845b1546e690f9369baa55e0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:47:40.406818 20673 raft_consensus.cc:729] T bab56985d291408caa7778c85def24fa P fd094c845b1546e690f9369baa55e0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd094c845b1546e690f9369baa55e0c2, State: Initialized, Role: FOLLOWER
I20260928 06:47:40.412168 20673 consensus_queue.cc:260] T bab56985d291408caa7778c85def24fa P fd094c845b1546e690f9369baa55e0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd094c845b1546e690f9369baa55e0c2" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 46177 } } peers { permanent_uuid: "88c89fa10457491187f58908b257489e" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35889 } } peers { permanent_uuid: "3cac415e9a4e4061bf487ff6ac3dae7a" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33093 } }
I20260928 06:47:40.412887 20637 heartbeater.cc:503] Master 127.7.48.62:45641 was elected leader, sending a full tablet report...
I20260928 06:47:40.413231 20673 ts_tablet_manager.cc:1428] T bab56985d291408caa7778c85def24fa P fd094c845b1546e690f9369baa55e0c2: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260928 06:47:40.414141 20674 log.cc:824] T bab56985d291408caa7778c85def24fa P 88c89fa10457491187f58908b257489e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:47:40.419726 20674 tablet_bootstrap.cc:492] T bab56985d291408caa7778c85def24fa P 88c89fa10457491187f58908b257489e: No bootstrap required, opened a new log
I20260928 06:47:40.419968 20674 ts_tablet_manager.cc:1397] T bab56985d291408caa7778c85def24fa P 88c89fa10457491187f58908b257489e: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.000s
I20260928 06:47:40.424080 20674 raft_consensus.cc:348] T bab56985d291408caa7778c85def24fa P 88c89fa10457491187f58908b257489e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd094c845b1546e690f9369baa55e0c2" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 46177 } } peers { permanent_uuid: "88c89fa10457491187f58908b257489e" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35889 } } peers { permanent_uuid: "3cac415e9a4e4061bf487ff6ac3dae7a" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33093 } }
I20260928 06:47:40.424376 20674 raft_consensus.cc:374] T bab56985d291408caa7778c85def24fa P 88c89fa10457491187f58908b257489e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:47:40.424476 20674 raft_consensus.cc:729] T bab56985d291408caa7778c85def24fa P 88c89fa10457491187f58908b257489e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88c89fa10457491187f58908b257489e, State: Initialized, Role: FOLLOWER
I20260928 06:47:40.424971 20674 consensus_queue.cc:260] T bab56985d291408caa7778c85def24fa P 88c89fa10457491187f58908b257489e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd094c845b1546e690f9369baa55e0c2" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 46177 } } peers { permanent_uuid: "88c89fa10457491187f58908b257489e" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35889 } } peers { permanent_uuid: "3cac415e9a4e4061bf487ff6ac3dae7a" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33093 } }
I20260928 06:47:40.425705 20369 heartbeater.cc:503] Master 127.7.48.62:45641 was elected leader, sending a full tablet report...
I20260928 06:47:40.426055 20674 ts_tablet_manager.cc:1428] T bab56985d291408caa7778c85def24fa P 88c89fa10457491187f58908b257489e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 06:47:40.451860 20678 raft_consensus.cc:482] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:47:40.452114 20678 raft_consensus.cc:504] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd094c845b1546e690f9369baa55e0c2" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 46177 } } peers { permanent_uuid: "88c89fa10457491187f58908b257489e" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35889 } } peers { permanent_uuid: "3cac415e9a4e4061bf487ff6ac3dae7a" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33093 } }
I20260928 06:47:40.453509 20678 leader_election.cc:288] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd094c845b1546e690f9369baa55e0c2 (127.7.48.3:46177), 88c89fa10457491187f58908b257489e (127.7.48.1:35889)
I20260928 06:47:40.460211 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:40.454332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:59889 (local address 127.7.48.1:35889)
0928 06:47:40.454478 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:40.454483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:40.455455 (+ 972us) server_negotiation.cc:407] Connection header received
0928 06:47:40.455668 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:40.455671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:40.455724 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:40.455791 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:47:40.456301 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:40.457070 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:40.458691 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:40.459303 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:40.459387 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:40.459460 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:40.459983 (+ 523us) server_negotiation.cc:299] Negotiation successful
0928 06:47:40.460067 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:47:40.460750 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:40.453931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:35889 (local address 127.7.48.2:59889)
0928 06:47:40.455343 (+ 1412us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:40.455357 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:40.455501 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:40.455986 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:40.455988 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:40.456010 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:47:40.456160 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:40.456169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:40.457221 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:40.457224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:40.458541 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:40.458555 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:40.459618 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:40.459621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:40.459796 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:40.459824 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 06:47:40.460483 (+ 659us) client_negotiation.cc:240] Negotiation successful
0928 06:47:40.460598 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1341,"mutex_wait_us":17,"thread_start_us":128,"threads_started":1}
I20260928 06:47:40.461416 20322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab56985d291408caa7778c85def24fa" candidate_uuid: "3cac415e9a4e4061bf487ff6ac3dae7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88c89fa10457491187f58908b257489e" is_pre_election: true
I20260928 06:47:40.461757 20322 raft_consensus.cc:2454] T bab56985d291408caa7778c85def24fa P 88c89fa10457491187f58908b257489e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cac415e9a4e4061bf487ff6ac3dae7a in term 0.
I20260928 06:47:40.462260 20390 leader_election.cc:302] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cac415e9a4e4061bf487ff6ac3dae7a, 88c89fa10457491187f58908b257489e; no voters:
I20260928 06:47:40.462555 20678 raft_consensus.cc:2781] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:47:40.462641 20678 raft_consensus.cc:482] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:47:40.462697 20678 raft_consensus.cc:3037] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:47:40.463500 20668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:40.455074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:39313 (local address 127.7.48.3:46177)
0928 06:47:40.455185 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:40.455189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:40.458403 (+ 3214us) server_negotiation.cc:407] Connection header received
0928 06:47:40.458666 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:40.458670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:40.458724 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:40.458773 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:47:40.459153 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:40.459865 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:40.462096 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:40.462659 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:40.462745 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:40.462807 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:40.463281 (+ 474us) server_negotiation.cc:299] Negotiation successful
0928 06:47:40.463353 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:47:40.464061 20682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:40.454694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:46177 (local address 127.7.48.2:39313)
0928 06:47:40.458307 (+ 3613us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:40.458322 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:40.458485 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:40.458907 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:40.458910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:40.458926 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:47:40.459030 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:40.459036 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:40.460034 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:40.460040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:40.461955 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:40.461965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:40.463054 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:40.463057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:40.463143 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:40.463168 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 06:47:40.463754 (+ 586us) client_negotiation.cc:240] Negotiation successful
0928 06:47:40.463819 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3548,"mutex_wait_us":3,"thread_start_us":107,"threads_started":1}
I20260928 06:47:40.464150 20678 raft_consensus.cc:504] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd094c845b1546e690f9369baa55e0c2" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 46177 } } peers { permanent_uuid: "88c89fa10457491187f58908b257489e" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35889 } } peers { permanent_uuid: "3cac415e9a4e4061bf487ff6ac3dae7a" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33093 } }
I20260928 06:47:40.464567 20678 leader_election.cc:288] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a [CANDIDATE]: Term 1 election: Requested vote from peers fd094c845b1546e690f9369baa55e0c2 (127.7.48.3:46177), 88c89fa10457491187f58908b257489e (127.7.48.1:35889)
I20260928 06:47:40.464778 20575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab56985d291408caa7778c85def24fa" candidate_uuid: "3cac415e9a4e4061bf487ff6ac3dae7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd094c845b1546e690f9369baa55e0c2" is_pre_election: true
I20260928 06:47:40.464973 20588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab56985d291408caa7778c85def24fa" candidate_uuid: "3cac415e9a4e4061bf487ff6ac3dae7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd094c845b1546e690f9369baa55e0c2"
I20260928 06:47:40.465081 20575 raft_consensus.cc:2454] T bab56985d291408caa7778c85def24fa P fd094c845b1546e690f9369baa55e0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cac415e9a4e4061bf487ff6ac3dae7a in term 0.
I20260928 06:47:40.465672 20322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab56985d291408caa7778c85def24fa" candidate_uuid: "3cac415e9a4e4061bf487ff6ac3dae7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88c89fa10457491187f58908b257489e"
I20260928 06:47:40.465794 20322 raft_consensus.cc:3037] T bab56985d291408caa7778c85def24fa P 88c89fa10457491187f58908b257489e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:47:40.466903 20322 raft_consensus.cc:2454] T bab56985d291408caa7778c85def24fa P 88c89fa10457491187f58908b257489e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cac415e9a4e4061bf487ff6ac3dae7a in term 1.
I20260928 06:47:40.467237 20390 leader_election.cc:302] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cac415e9a4e4061bf487ff6ac3dae7a, 88c89fa10457491187f58908b257489e; no voters: fd094c845b1546e690f9369baa55e0c2
I20260928 06:47:40.467480 20678 raft_consensus.cc:2781] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:47:40.467710 20678 raft_consensus.cc:686] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a [term 1 LEADER]: Becoming Leader. State: Replica: 3cac415e9a4e4061bf487ff6ac3dae7a, State: Running, Role: LEADER
I20260928 06:47:40.468071 20678 consensus_queue.cc:237] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd094c845b1546e690f9369baa55e0c2" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 46177 } } peers { permanent_uuid: "88c89fa10457491187f58908b257489e" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35889 } } peers { permanent_uuid: "3cac415e9a4e4061bf487ff6ac3dae7a" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33093 } }
I20260928 06:47:40.471426 20171 catalog_manager.cc:5495] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cac415e9a4e4061bf487ff6ac3dae7a (127.7.48.2). New cstate: current_term: 1 leader_uuid: "3cac415e9a4e4061bf487ff6ac3dae7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd094c845b1546e690f9369baa55e0c2" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 46177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88c89fa10457491187f58908b257489e" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cac415e9a4e4061bf487ff6ac3dae7a" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33093 } health_report { overall_health: HEALTHY } } }
W20260928 06:47:40.473485 20370 tablet.cc:2369] T bab56985d291408caa7778c85def24fa P 88c89fa10457491187f58908b257489e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:47:40.474251 20504 tablet.cc:2369] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:47:40.484366 20638 tablet.cc:2369] T bab56985d291408caa7778c85def24fa P fd094c845b1546e690f9369baa55e0c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:47:40.510457 20672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:40.507309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35992 (local address 127.7.48.2:33093)
0928 06:47:40.507441 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:40.507445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:40.507457 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:47:40.507537 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:40.507539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:40.507591 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:40.507698 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:47:40.507976 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:40.508467 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:40.509236 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:40.509431 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:40.509508 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:40.509737 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:47:40.510128 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:47:40.510217 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:40.510265 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 06:47:40.510313 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 06:47:40.515997 20588 raft_consensus.cc:3037] T bab56985d291408caa7778c85def24fa P fd094c845b1546e690f9369baa55e0c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:47:40.516124 20322 raft_consensus.cc:1261] T bab56985d291408caa7778c85def24fa P 88c89fa10457491187f58908b257489e [term 1 FOLLOWER]: Refusing update from remote peer 3cac415e9a4e4061bf487ff6ac3dae7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:47:40.516860 20678 consensus_queue.cc:1035] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "88c89fa10457491187f58908b257489e" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 35889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:47:40.517537 20588 raft_consensus.cc:1261] T bab56985d291408caa7778c85def24fa P fd094c845b1546e690f9369baa55e0c2 [term 1 FOLLOWER]: Refusing update from remote peer 3cac415e9a4e4061bf487ff6ac3dae7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:47:40.518082 20678 consensus_queue.cc:1035] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd094c845b1546e690f9369baa55e0c2" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 46177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:47:40.524413 20695 mvcc.cc:205] Tried to move back new op lower bound from 7334207735866490880 to 7334207735680208896. Current Snapshot: MvccSnapshot[applied={T|T < 7334207735866490880}]
I20260928 06:47:40.528762 20694 mvcc.cc:205] Tried to move back new op lower bound from 7334207735866490880 to 7334207735680208896. Current Snapshot: MvccSnapshot[applied={T|T < 7334207735866490880}]
I20260928 06:47:40.536170 20170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:47:40.536345 20170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:47:40.538618 20170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3c8888d139a4ff18f9431153b01dd0f: Sending DeleteTablet for 3 replicas of tablet bab56985d291408caa7778c85def24fa
I20260928 06:47:40.539286 20437 tablet_service.cc:1514] Processing DeleteTablet for tablet bab56985d291408caa7778c85def24fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:47:40 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:35976
I20260928 06:47:40.539289 20303 tablet_service.cc:1514] Processing DeleteTablet for tablet bab56985d291408caa7778c85def24fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:47:40 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:33944
I20260928 06:47:40.539741 20704 tablet_replica.cc:330] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a: stopping tablet replica
I20260928 06:47:40.539742 20571 tablet_service.cc:1514] Processing DeleteTablet for tablet bab56985d291408caa7778c85def24fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:47:40 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:43780
I20260928 06:47:40.540055 20704 raft_consensus.cc:2229] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:47:40.540233 20706 tablet_replica.cc:330] T bab56985d291408caa7778c85def24fa P fd094c845b1546e690f9369baa55e0c2: stopping tablet replica
I20260928 06:47:40.540395 20704 raft_consensus.cc:2258] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:47:40.540597 20706 raft_consensus.cc:2229] T bab56985d291408caa7778c85def24fa P fd094c845b1546e690f9369baa55e0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:47:40.540853 20706 raft_consensus.cc:2258] T bab56985d291408caa7778c85def24fa P fd094c845b1546e690f9369baa55e0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:47:40.541080 20705 tablet_replica.cc:330] T bab56985d291408caa7778c85def24fa P 88c89fa10457491187f58908b257489e: stopping tablet replica
I20260928 06:47:40.541237 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 20209
I20260928 06:47:40.541333 20705 raft_consensus.cc:2229] T bab56985d291408caa7778c85def24fa P 88c89fa10457491187f58908b257489e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:47:40.541599 20705 raft_consensus.cc:2258] T bab56985d291408caa7778c85def24fa P 88c89fa10457491187f58908b257489e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:47:40.544193 20704 ts_tablet_manager.cc:1905] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:47:40.545310 20706 ts_tablet_manager.cc:1905] T bab56985d291408caa7778c85def24fa P fd094c845b1546e690f9369baa55e0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:47:40.547773 20706 ts_tablet_manager.cc:1918] T bab56985d291408caa7778c85def24fa P fd094c845b1546e690f9369baa55e0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:47:40.547894 20706 log.cc:1196] T bab56985d291408caa7778c85def24fa P fd094c845b1546e690f9369baa55e0c2: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/bab56985d291408caa7778c85def24fa
I20260928 06:47:40.547827 20704 ts_tablet_manager.cc:1918] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:47:40.548166 20704 log.cc:1196] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/bab56985d291408caa7778c85def24fa
I20260928 06:47:40.548264 20706 ts_tablet_manager.cc:1939] T bab56985d291408caa7778c85def24fa P fd094c845b1546e690f9369baa55e0c2: Deleting consensus metadata
I20260928 06:47:40.548630 20704 ts_tablet_manager.cc:1939] T bab56985d291408caa7778c85def24fa P 3cac415e9a4e4061bf487ff6ac3dae7a: Deleting consensus metadata
I20260928 06:47:40.549876 20157 catalog_manager.cc:4841] TS 3cac415e9a4e4061bf487ff6ac3dae7a (127.7.48.2:33093): tablet bab56985d291408caa7778c85def24fa (table test-table [id=c5aaf6838d154528a16b4ef105843e2e]) successfully deleted
W20260928 06:47:40.550071 20155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.48.1:35889 (error 108)
I20260928 06:47:40.549877 20158 catalog_manager.cc:4841] TS fd094c845b1546e690f9369baa55e0c2 (127.7.48.3:46177): tablet bab56985d291408caa7778c85def24fa (table test-table [id=c5aaf6838d154528a16b4ef105843e2e]) successfully deleted
I20260928 06:47:40.550896 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 20373
I20260928 06:47:40.551115 20667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:40.550801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:35889 (local address 127.0.0.1:33958)
0928 06:47:40.550923 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:40.551014 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.1:35889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:47:40.551584 20155 catalog_manager.cc:4584] TS 88c89fa10457491187f58908b257489e (127.7.48.1:35889): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bab56985d291408caa7778c85def24fa: Network error: Client connection negotiation failed: client connection to 127.7.48.1:35889: connect: Connection refused (error 111)
I20260928 06:47:40.556818 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 20506
I20260928 06:47:40.562455 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 20138
I20260928 06:47:40.570503 7360 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 06:47:40.904734 7360 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 06:47:40.904838 7360 mini_ranger.cc:66] Stopping Ranger...
I20260928 06:47:41.250267 7360 mini_ranger.cc:68] Stopped Ranger
2026-09-28 06:47:41.250 UTC [19205] LOG: received smart shutdown request
2026-09-28 06:47:41.254 UTC [19205] LOG: background worker "logical replication launcher" (PID 19214) exited with exit code 1
2026-09-28 06:47:41.254 UTC [19209] LOG: shutting down
2026-09-28 06:47:41.411 UTC [19205] LOG: database system is shut down
2026-09-28T06:47:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45868 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:47:41 dist-test-slave-twl6 krb5kdc[20716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:47:41 dist-test-slave-twl6 krb5kdc[20716](info): set up 2 sockets
Sep 28 06:47:41 dist-test-slave-twl6 krb5kdc[20716](info): commencing operation
krb5kdc: starting...
W20260928 06:47:42.521520 7360 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:47:42 dist-test-slave-twl6 krb5kdc[20716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:47:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:47:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:47:42.634049 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:36655
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:36655,127.7.48.61:36723,127.7.48.60:45337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:47:42.740593 20729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:47:42.740867 20729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:47:42.740919 20729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:47:42.745008 20729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:47:42.745066 20729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:47:42.745086 20729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:47:42.745103 20729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:47:42.745118 20729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:47:42.750830 20729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:36655,127.7.48.61:36723,127.7.48.60:45337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:36655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:47:42.751799 20729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:47:42.752866 20729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:47:42.761107 20734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:42.761227 20729 server_base.cc:1034] running on GCE node
W20260928 06:47:42.761116 20737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:42.761122 20735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:42.761884 20729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:47:42.762919 20729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:47:42.764101 20729 hybrid_clock.cc:648] HybridClock initialized: now 1790578062764080 us; error 39 us; skew 500 ppm
Sep 28 06:47:42 dist-test-slave-twl6 krb5kdc[20716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578062, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:42.766671 20729 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:47:42.767800 20729 webserver.cc:460] Webserver started at http://127.7.48.62:40963/ using document root <none> and password file <none>
I20260928 06:47:42.768529 20729 fs_manager.cc:359] Metadata directory not provided
I20260928 06:47:42.768637 20729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:47:42.768862 20729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:47:42.771275 20729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "bb8335216a7641a89a55faab75ad922c"
format_stamp: "Formatted at 2026-09-28 06:47:42 on dist-test-slave-twl6"
server_key: "f874d33b47bc922a786380f53f96bdd5"
server_key_iv: "5fddffe1011011e75b5c3ac2788cbe98"
server_key_version: "encryptionkey@0"
I20260928 06:47:42.771987 20729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "bb8335216a7641a89a55faab75ad922c"
format_stamp: "Formatted at 2026-09-28 06:47:42 on dist-test-slave-twl6"
server_key: "f874d33b47bc922a786380f53f96bdd5"
server_key_iv: "5fddffe1011011e75b5c3ac2788cbe98"
server_key_version: "encryptionkey@0"
I20260928 06:47:42.776558 20729 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 06:47:42.779249 20744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:42.780442 20729 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:47:42.780558 20729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "bb8335216a7641a89a55faab75ad922c"
format_stamp: "Formatted at 2026-09-28 06:47:42 on dist-test-slave-twl6"
server_key: "f874d33b47bc922a786380f53f96bdd5"
server_key_iv: "5fddffe1011011e75b5c3ac2788cbe98"
server_key_version: "encryptionkey@0"
I20260928 06:47:42.780639 20729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:47:42.798241 20729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:47:42.801077 20729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:47:42.801223 20729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:47:42.808073 20729 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:36655
I20260928 06:47:42.808151 20796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:36655 every 8 connection(s)
I20260928 06:47:42.809041 20729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:47:42.811689 20797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:47:42.814580 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 20729
I20260928 06:47:42.814729 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:47:42.814985 7360 external_mini_cluster.cc:1428] Setting key d25ef9116d96b8005249aadf15bc97ff
I20260928 06:47:42.815354 20797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36655 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.48.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.61@KRBTEST.COM" created.
Sep 28 06:47:42 dist-test-slave-twl6 krb5kdc[20716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578062, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:47:42.830636 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:42.817691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:36655 (local address 127.0.0.1:53326)
0928 06:47:42.818261 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:42.818306 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:42.819059 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:42.819799 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:42.819803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:42.820139 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:47:42.820378 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:42.820390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:42.821935 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:42.821938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:42.822660 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:42.822669 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:42.823230 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:42.823232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:42.823319 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:42.823790 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:47:42.823804 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:47:42.825743 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:47:42.827939 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:42.827946 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:42.827953 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:42.828154 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:42.828528 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:42.828531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:42.828533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:42.828657 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:42.829373 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:47:42.829378 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:47:42.829486 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 06:47:42.829611 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 06:47:42.829838 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"spinlock_wait_cycles":49536,"thread_start_us":135,"threads_started":1}
I20260928 06:47:42.830647 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:42.817691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53326 (local address 127.7.48.62:36655)
0928 06:47:42.818161 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:42.818180 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:42.818423 (+ 243us) server_negotiation.cc:407] Connection header received
0928 06:47:42.819364 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:42.819372 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:42.819598 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:42.819802 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:42.820524 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:42.821823 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:42.822772 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:42.823096 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:42.823213 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:42.825900 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:42.825904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:42.825913 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:42.825930 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:42.827770 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:42.828279 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:42.828287 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:42.828292 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:42.828425 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:42.828762 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:42.828767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:42.828770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:42.829262 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:42.829402 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:42.829740 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 06:47:42.829905 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":134,"threads_started":1}
I20260928 06:47:42.832965 20797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:47:42.833606 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:42.833344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:36723 (local address 127.0.0.1:55430)
0928 06:47:42.833484 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:42.833556 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.61:36723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 06:47:42.833844 20747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.48.61:36723: connect: Connection refused (error 111)
I20260928 06:47:42.834714 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:42.834484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:36723 (local address 127.0.0.1:55438)
0928 06:47:42.834599 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:42.834668 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.61:36723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 06:47:42.834865 20797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.61:36723: Network error: Client connection negotiation failed: client connection to 127.7.48.61:36723: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.61.keytab.
Entry for principal kudu/127.7.48.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.61.keytab.
I20260928 06:47:42.840214 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.61:36723
--webserver_interface=127.7.48.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.61.keytab
--principal=kudu/127.7.48.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:36655,127.7.48.61:36723,127.7.48.60:45337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:47:42.884105 20797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 } attempt: 1
I20260928 06:47:42.884958 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:42.884669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:36723 (local address 127.0.0.1:55454)
0928 06:47:42.884812 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:42.884884 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.61:36723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 06:47:42.885680 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:42.885474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:36723 (local address 127.0.0.1:55462)
0928 06:47:42.885581 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:42.885630 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.61:36723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 06:47:42.885811 20797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.61:36723: Network error: Client connection negotiation failed: client connection to 127.7.48.61:36723: connect: Connection refused (error 111)
W20260928 06:47:42.945214 20804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:47:42.945505 20804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:47:42.945591 20804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:47:42.949389 20804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:47:42.949484 20804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:47:42.949551 20804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:47:42.949581 20804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:47:42.949611 20804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:47:42.954001 20797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 } attempt: 2
I20260928 06:47:42.954787 20804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:36655,127.7.48.61:36723,127.7.48.60:45337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.61.keytab
--principal=kudu/127.7.48.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.61:36723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.48.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:47:42.954895 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:42.954595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:36723 (local address 127.0.0.1:55464)
0928 06:47:42.954738 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:42.954807 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.61:36723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 06:47:42.955684 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:42.955412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:36723 (local address 127.0.0.1:55468)
0928 06:47:42.955558 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:42.955619 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.61:36723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 06:47:42.955850 20797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.61:36723: Network error: Client connection negotiation failed: client connection to 127.7.48.61:36723: connect: Connection refused (error 111)
I20260928 06:47:42.956034 20804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:47:42.956928 20804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:47:42.965030 20809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:42.965032 20810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:42.965049 20812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:42.965255 20804 server_base.cc:1034] running on GCE node
I20260928 06:47:42.965919 20804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:47:42.966930 20804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:47:42.968102 20804 hybrid_clock.cc:648] HybridClock initialized: now 1790578062968082 us; error 40 us; skew 500 ppm
Sep 28 06:47:42 dist-test-slave-twl6 krb5kdc[20716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578062, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:42.970731 20804 init.cc:377] Logged in from keytab as kudu/127.7.48.61@KRBTEST.COM (short username kudu)
I20260928 06:47:42.971555 20804 webserver.cc:460] Webserver started at http://127.7.48.61:38873/ using document root <none> and password file <none>
I20260928 06:47:42.972136 20804 fs_manager.cc:359] Metadata directory not provided
I20260928 06:47:42.972222 20804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:47:42.972438 20804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:47:42.974362 20804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-1/data/instance:
uuid: "c4b573c7762c41e5b18139dd2a3b7fad"
format_stamp: "Formatted at 2026-09-28 06:47:42 on dist-test-slave-twl6"
server_key: "b426f366e3823c3fc0f4955f4752806a"
server_key_iv: "7d6d6a978d8c25f396a8009e93514307"
server_key_version: "encryptionkey@0"
I20260928 06:47:42.974967 20804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-1/wal/instance:
uuid: "c4b573c7762c41e5b18139dd2a3b7fad"
format_stamp: "Formatted at 2026-09-28 06:47:42 on dist-test-slave-twl6"
server_key: "b426f366e3823c3fc0f4955f4752806a"
server_key_iv: "7d6d6a978d8c25f396a8009e93514307"
server_key_version: "encryptionkey@0"
I20260928 06:47:42.978516 20804 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 06:47:42.980702 20819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:42.981768 20804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:47:42.981895 20804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-1/wal
uuid: "c4b573c7762c41e5b18139dd2a3b7fad"
format_stamp: "Formatted at 2026-09-28 06:47:42 on dist-test-slave-twl6"
server_key: "b426f366e3823c3fc0f4955f4752806a"
server_key_iv: "7d6d6a978d8c25f396a8009e93514307"
server_key_version: "encryptionkey@0"
I20260928 06:47:42.981989 20804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:47:43.005849 20804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:47:43.009218 20804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:47:43.009445 20804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:47:43.016571 20804 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.61:36723
I20260928 06:47:43.016589 20871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.61:36723 every 8 connection(s)
I20260928 06:47:43.017472 20804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-1/data/info.pb
I20260928 06:47:43.020004 20872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:47:43.023391 20872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36655 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:47:43.026100 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 20804
I20260928 06:47:43.026229 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-1/wal/instance
I20260928 06:47:43.026463 7360 external_mini_cluster.cc:1428] Setting key 9e0cd94cc9a81615eadebf756d78aa40
Sep 28 06:47:43 dist-test-slave-twl6 krb5kdc[20716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578062, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.61@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.48.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.60@KRBTEST.COM" created.
I20260928 06:47:43.036841 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.025214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53342 (local address 127.7.48.62:36655)
0928 06:47:43.025348 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:43.025351 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:43.026061 (+ 710us) server_negotiation.cc:407] Connection header received
0928 06:47:43.026993 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:43.026996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:43.027052 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:43.027145 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:43.028210 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.028699 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.029702 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.029923 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.030026 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:43.032750 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:43.032752 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:43.032754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:43.032775 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:43.034861 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.035348 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.035350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.035352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.035424 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.035725 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.035727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.035728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.035997 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:43.036074 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:43.036564 (+ 490us) server_negotiation.cc:299] Negotiation successful
0928 06:47:43.036710 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:47:43.037933 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.025491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:36655 (local address 127.0.0.1:53342)
0928 06:47:43.025923 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:43.025961 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:43.026812 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:43.027301 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:43.027307 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:43.027589 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:47:43.028077 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:43.028090 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.028829 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.028832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:43.029589 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:43.029597 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.030062 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.030066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:43.030228 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:43.030890 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:47:43.030905 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:47:43.032594 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:47:43.034982 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:43.034987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:43.035009 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:43.035256 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:43.035509 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:43.035511 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:43.035512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:43.035645 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:43.036087 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:47:43.036091 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:47:43.036317 (+ 226us) client_negotiation.cc:769] Sending connection context
0928 06:47:43.036469 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 06:47:43.036822 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":114,"threads_started":1}
I20260928 06:47:43.039402 20872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:47:43 dist-test-slave-twl6 krb5kdc[20716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578062, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.61@KRBTEST.COM for kudu/127.7.48.61@KRBTEST.COM
I20260928 06:47:43.047011 20797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 } attempt: 3
I20260928 06:47:43.051237 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.039983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:36723 (local address 127.0.0.1:55482)
0928 06:47:43.040113 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:43.040128 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:43.040202 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:43.040832 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:43.040838 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:43.041019 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:47:43.041158 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:43.041167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.042118 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.042121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:43.042998 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:43.043012 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.043595 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.043602 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:43.043739 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:43.044218 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:47:43.044232 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:47:43.047259 (+ 3027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:47:43.049371 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:43.049374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:43.049375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:43.049547 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:43.049853 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:43.049855 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:43.049856 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:43.049955 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:43.050491 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:47:43.050493 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:47:43.050582 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 06:47:43.050779 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 06:47:43.050898 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"spinlock_wait_cycles":41728}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.60.keytab.
Entry for principal kudu/127.7.48.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.60.keytab.
I20260928 06:47:43.051666 20877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.040163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.7.48.61:36723)
0928 06:47:43.040499 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:43.040508 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:43.040525 (+ 17us) server_negotiation.cc:407] Connection header received
0928 06:47:43.040557 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:43.040560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:43.040698 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:43.040795 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:43.041286 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.042006 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.043113 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.043479 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.043558 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:43.047419 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:43.047430 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:43.047461 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:43.047503 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:43.049266 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.049651 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.049657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.049659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.049745 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.050054 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.050057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.050058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.050400 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:43.050498 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:43.050906 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 06:47:43.051060 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":169,"threads_started":1}
Sep 28 06:47:43 dist-test-slave-twl6 krb5kdc[20716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578062, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for kudu/127.7.48.61@KRBTEST.COM
I20260928 06:47:43.052654 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.60:45337
--webserver_interface=127.7.48.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.60.keytab
--principal=kudu/127.7.48.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:36655,127.7.48.61:36723,127.7.48.60:45337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:47:43.053972 20872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:47:43.054616 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.054392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:45337 (local address 127.0.0.1:49266)
0928 06:47:43.054488 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:43.054564 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 06:47:43.054875 20821 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.48.60:45337: connect: Connection refused (error 111)
I20260928 06:47:43.055807 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.055446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:45337 (local address 127.0.0.1:49270)
0928 06:47:43.055710 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:43.055749 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213}
W20260928 06:47:43.055976 20872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.60:45337: Network error: Client connection negotiation failed: client connection to 127.7.48.60:45337: connect: Connection refused (error 111)
I20260928 06:47:43.056659 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.047451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:36723 (local address 127.0.0.1:55494)
0928 06:47:43.047594 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:43.047610 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:43.047690 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:43.048089 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:43.048094 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:43.048264 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:47:43.048394 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:43.048401 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.049256 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.049259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:43.050060 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:43.050069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.050597 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.050599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:43.050748 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:43.051229 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:47:43.051243 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:47:43.052710 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:47:43.054819 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:43.054823 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:43.054825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:43.055090 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:43.055456 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:43.055462 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:43.055464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:43.055564 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:43.056085 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:47:43.056090 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:47:43.056199 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 06:47:43.056282 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 06:47:43.056428 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 06:47:43.057395 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.047507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55494 (local address 127.7.48.61:36723)
0928 06:47:43.047849 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:43.047859 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:43.047891 (+ 32us) server_negotiation.cc:407] Connection header received
0928 06:47:43.047926 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:43.047930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:43.047972 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:43.048050 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:43.048541 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.049104 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.050181 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.050486 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.050575 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:43.052832 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:43.052836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:43.052838 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:43.052859 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:43.054680 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.055229 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.055234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.055237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.055326 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.055672 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.055678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.055682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.055971 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:43.056956 (+ 985us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:43.057034 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 06:47:43.057167 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"spinlock_wait_cycles":1408,"thread_start_us":124,"threads_started":1}
I20260928 06:47:43.058064 20797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:47:43.058823 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.058503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:45337 (local address 127.0.0.1:49276)
0928 06:47:43.058690 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:43.058743 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260928 06:47:43.059535 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.059233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:45337 (local address 127.0.0.1:49278)
0928 06:47:43.059396 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:43.059453 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260928 06:47:43.059687 20797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.60:45337: Network error: Client connection negotiation failed: client connection to 127.7.48.60:45337: connect: Connection refused (error 111)
I20260928 06:47:43.090854 20797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } attempt: 1
I20260928 06:47:43.091698 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.091331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:45337 (local address 127.0.0.1:49282)
0928 06:47:43.091537 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:43.091608 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 06:47:43.092413 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.092200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:45337 (local address 127.0.0.1:49296)
0928 06:47:43.092304 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:43.092356 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 06:47:43.092561 20797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.60:45337: Network error: Client connection negotiation failed: client connection to 127.7.48.60:45337: connect: Connection refused (error 111)
I20260928 06:47:43.105230 20872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } attempt: 1
I20260928 06:47:43.105952 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.105683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:45337 (local address 127.0.0.1:49302)
0928 06:47:43.105806 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:43.105890 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 06:47:43.106629 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.106442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:45337 (local address 127.0.0.1:49310)
0928 06:47:43.106536 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:43.106576 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 06:47:43.106787 20872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.60:45337: Network error: Client connection negotiation failed: client connection to 127.7.48.60:45337: connect: Connection refused (error 111)
W20260928 06:47:43.158520 20879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:47:43.158766 20879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:47:43.158825 20879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:47:43.162580 20879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:47:43.162673 20879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:47:43.162698 20879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:47:43.162770 20879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:47:43.162801 20879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:47:43.167732 20797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } attempt: 2
I20260928 06:47:43.168303 20879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:36655,127.7.48.61:36723,127.7.48.60:45337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.60.keytab
--principal=kudu/127.7.48.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.60:45337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.48.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:47:43.168615 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.168330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:45337 (local address 127.0.0.1:49324)
0928 06:47:43.168464 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:43.168529 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 06:47:43.169353 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.169141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:45337 (local address 127.0.0.1:49330)
0928 06:47:43.169253 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:43.169298 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 06:47:43.169471 20797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.60:45337: Network error: Client connection negotiation failed: client connection to 127.7.48.60:45337: connect: Connection refused (error 111)
I20260928 06:47:43.169503 20879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:47:43.170452 20879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:47:43.174930 20872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } attempt: 2
I20260928 06:47:43.175598 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.175364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:45337 (local address 127.0.0.1:49346)
0928 06:47:43.175484 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:43.175536 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 06:47:43.176252 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.176060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:45337 (local address 127.0.0.1:49350)
0928 06:47:43.176160 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:43.176201 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 06:47:43.176360 20872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.60:45337: Network error: Client connection negotiation failed: client connection to 127.7.48.60:45337: connect: Connection refused (error 111)
W20260928 06:47:43.178862 20886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:43.178877 20885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:43.178921 20888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:43.179149 20879 server_base.cc:1034] running on GCE node
I20260928 06:47:43.179718 20879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:47:43.180733 20879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:47:43.181902 20879 hybrid_clock.cc:648] HybridClock initialized: now 1790578063181887 us; error 27 us; skew 500 ppm
Sep 28 06:47:43 dist-test-slave-twl6 krb5kdc[20716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578063, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:43.184512 20879 init.cc:377] Logged in from keytab as kudu/127.7.48.60@KRBTEST.COM (short username kudu)
I20260928 06:47:43.185406 20879 webserver.cc:460] Webserver started at http://127.7.48.60:34647/ using document root <none> and password file <none>
I20260928 06:47:43.185967 20879 fs_manager.cc:359] Metadata directory not provided
I20260928 06:47:43.186056 20879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:47:43.186291 20879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:47:43.188195 20879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-2/data/instance:
uuid: "16219647b37942c5b45f3b595fccc0c5"
format_stamp: "Formatted at 2026-09-28 06:47:43 on dist-test-slave-twl6"
server_key: "e27c8704aab0c09d356cac6ef9db7555"
server_key_iv: "995b09b830b16c18d18780b9e1fda6e4"
server_key_version: "encryptionkey@0"
I20260928 06:47:43.188705 20879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-2/wal/instance:
uuid: "16219647b37942c5b45f3b595fccc0c5"
format_stamp: "Formatted at 2026-09-28 06:47:43 on dist-test-slave-twl6"
server_key: "e27c8704aab0c09d356cac6ef9db7555"
server_key_iv: "995b09b830b16c18d18780b9e1fda6e4"
server_key_version: "encryptionkey@0"
I20260928 06:47:43.192123 20879 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:47:43.194299 20895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:43.195472 20879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:47:43.195608 20879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-2/wal
uuid: "16219647b37942c5b45f3b595fccc0c5"
format_stamp: "Formatted at 2026-09-28 06:47:43 on dist-test-slave-twl6"
server_key: "e27c8704aab0c09d356cac6ef9db7555"
server_key_iv: "995b09b830b16c18d18780b9e1fda6e4"
server_key_version: "encryptionkey@0"
I20260928 06:47:43.195713 20879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:47:43.205435 20879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:47:43.207996 20879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:47:43.208181 20879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:47:43.214574 20879 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.60:45337
I20260928 06:47:43.214603 20947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.60:45337 every 8 connection(s)
I20260928 06:47:43.215471 20879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-2/data/info.pb
I20260928 06:47:43.217984 20948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:47:43.219235 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 20879
I20260928 06:47:43.219363 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577905526996-7360-0/minicluster-data/master-2/wal/instance
I20260928 06:47:43.219614 7360 external_mini_cluster.cc:1428] Setting key c856ad2e809aeab71f468644d3f15f7f
I20260928 06:47:43.221982 20948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36655 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:47:43 dist-test-slave-twl6 krb5kdc[20716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.60@KRBTEST.COM
Sep 28 06:47:43 dist-test-slave-twl6 krb5kdc[20716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578063, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.60@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:47:43.231916 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.223497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53350 (local address 127.7.48.62:36655)
0928 06:47:43.223642 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:43.223646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:43.223875 (+ 229us) server_negotiation.cc:407] Connection header received
0928 06:47:43.224006 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:43.224009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:43.224052 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:43.224125 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:43.224740 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.225238 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.226194 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.226379 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.226471 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:43.228873 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:43.228875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:43.228877 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:43.228898 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:43.230233 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.230680 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.230682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.230683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.230745 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.230986 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.230988 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.230989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.231184 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:43.231285 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:43.231686 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 06:47:43.231772 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 06:47:43.232555 20951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.220873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49354 (local address 127.7.48.60:45337)
0928 06:47:43.221264 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:43.221278 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:43.221309 (+ 31us) server_negotiation.cc:407] Connection header received
0928 06:47:43.221866 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:43.221880 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:43.222161 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:43.222455 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:43.223113 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.224383 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.225232 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.225737 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.225915 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:43.228337 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:43.228342 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:43.228353 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:43.228376 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:43.229760 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.230175 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.230180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.230184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.230296 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.230539 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.230542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.230544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.230958 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:43.231108 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:43.231401 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 06:47:43.231661 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":273,"spinlock_wait_cycles":2304,"thread_start_us":132,"threads_started":1}
I20260928 06:47:43.232553 20952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.223368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:36655 (local address 127.0.0.1:53350)
0928 06:47:43.223747 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:43.223790 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:43.223926 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:43.224177 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:43.224181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:43.224462 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:47:43.224583 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:43.224588 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.225365 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.225369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:43.226065 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:43.226074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.226530 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.226534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:43.226622 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:43.227112 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:47:43.227125 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:47:43.228717 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:47:43.230323 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:43.230333 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:43.230340 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:43.230597 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:43.230821 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:43.230824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:43.230826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:43.230912 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:43.231294 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:47:43.231300 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:47:43.231403 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 06:47:43.231576 (+ 173us) client_negotiation.cc:240] Negotiation successful
0928 06:47:43.231705 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":48,"spinlock_wait_cycles":384,"thread_start_us":142,"threads_started":1}
I20260928 06:47:43.233681 20948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:47:43 dist-test-slave-twl6 krb5kdc[20716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578063, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.60@KRBTEST.COM for kudu/127.7.48.61@KRBTEST.COM
Sep 28 06:47:43 dist-test-slave-twl6 krb5kdc[20716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:47:43.241683 20952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.234187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:36723 (local address 127.0.0.1:55504)
0928 06:47:43.234317 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:43.234332 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:43.234423 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:43.234830 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:43.234832 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:43.234950 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:47:43.235031 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:43.235036 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.235798 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.235801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:43.236338 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:43.236346 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.236810 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.236813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:43.236897 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:43.237313 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:47:43.237329 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:47:43.238446 (+ 1117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:47:43.240311 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:43.240313 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:43.240314 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:43.240477 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:43.240780 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:43.240782 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:43.240784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:43.240872 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:43.241277 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:47:43.241279 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:47:43.241367 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 06:47:43.241471 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 06:47:43.241573 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 06:47:43.241809 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.234314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55504 (local address 127.7.48.61:36723)
0928 06:47:43.234460 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:43.234467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:43.234483 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:47:43.234534 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:43.234537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:43.234610 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:43.234703 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:43.235155 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.235662 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.236456 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.236705 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.236810 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:43.238583 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:43.238586 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:43.238588 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:43.238608 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:43.240213 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.240578 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.240581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.240583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.240664 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.240960 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.240962 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.240963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.241170 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:43.241263 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:43.241528 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 06:47:43.241651 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 06:47:43.242475 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.234689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53356 (local address 127.7.48.62:36655)
0928 06:47:43.234795 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:43.234800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:43.234814 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:47:43.234874 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:43.234877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:43.234912 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:43.235002 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:43.235544 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.236075 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.236802 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.236988 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.237083 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:43.239228 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:43.239232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:43.239234 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:43.239257 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:43.240688 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.241103 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.241106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.241108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.241177 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.241472 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.241475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.241478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.241694 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:43.241892 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:43.242229 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 06:47:43.242341 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:47:43.242559 20948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:47:43 dist-test-slave-twl6 krb5kdc[20716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578063, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.60@KRBTEST.COM for kudu/127.7.48.60@KRBTEST.COM
Sep 28 06:47:43 dist-test-slave-twl6 krb5kdc[20716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.61@KRBTEST.COM
I20260928 06:47:43.251178 20952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.242960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:45337 (local address 127.0.0.1:49364)
0928 06:47:43.243066 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:43.243079 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:43.243138 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:43.243518 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:43.243520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:43.243665 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:47:43.243752 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:43.243758 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.245097 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.245100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:43.245663 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:43.245671 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.246189 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.246193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:43.246282 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:43.246713 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:47:43.246730 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:47:43.247839 (+ 1109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:47:43.249591 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:43.249593 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:43.249594 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:43.249760 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:43.250279 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:43.250281 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:43.250282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:43.250347 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:43.250738 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:47:43.250740 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:47:43.250802 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 06:47:43.250905 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 06:47:43.251035 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 06:47:43.251412 20951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.243041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49364 (local address 127.7.48.60:45337)
0928 06:47:43.243128 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:43.243130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:43.243140 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:47:43.243377 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:43.243379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:43.243410 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:43.243504 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:43.243864 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.244996 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.245772 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.246078 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.246158 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:43.247985 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:43.247987 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:43.247989 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:43.248008 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:43.249488 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.249854 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.249857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.249860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.249933 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.250478 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.250480 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.250481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.250648 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:43.250732 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:43.250972 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 06:47:43.251099 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":33}
I20260928 06:47:43.251473 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.243661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55512 (local address 127.7.48.61:36723)
0928 06:47:43.243784 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:43.243790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:43.243804 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:47:43.243839 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:43.243842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:43.243900 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:43.243977 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:43.244690 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.245200 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.245956 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.246186 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.246284 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:43.248525 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:43.248530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:43.248534 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:43.248560 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:43.249932 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.250263 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.250265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.250266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.250325 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.250607 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.250610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.250611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.250796 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:43.250950 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:43.251112 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 06:47:43.251198 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 06:47:43.254503 20948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5: Bootstrap starting.
I20260928 06:47:43.254552 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:47:43.257171 20948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5: Neither blocks nor log segments found. Creating new log.
I20260928 06:47:43.258031 20948 log.cc:824] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5: Log is configured to *not* fsync() on all Append() calls
I20260928 06:47:43.261060 20948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5: No bootstrap required, opened a new log
I20260928 06:47:43.265564 20948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8335216a7641a89a55faab75ad922c" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36655 } } peers { permanent_uuid: "c4b573c7762c41e5b18139dd2a3b7fad" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 } } peers { permanent_uuid: "16219647b37942c5b45f3b595fccc0c5" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } }
I20260928 06:47:43.265961 20948 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:47:43.266172 20951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.256852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49368 (local address 127.7.48.60:45337)
0928 06:47:43.256944 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:43.256948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:43.257088 (+ 140us) server_negotiation.cc:407] Connection header received
0928 06:47:43.257225 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:43.257228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:43.257262 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:43.257318 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:43.258130 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.258908 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.260093 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.260409 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.260473 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:43.262285 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:43.262289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:43.262291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:43.262314 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:43.264132 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.264661 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.264689 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.264693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.264784 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.265135 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.265138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.265140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.265387 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:43.265482 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:43.265844 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 06:47:43.265967 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:47:43.266445 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.256531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53368 (local address 127.7.48.62:36655)
0928 06:47:43.256621 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:43.256625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:43.257209 (+ 584us) server_negotiation.cc:407] Connection header received
0928 06:47:43.257613 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:43.257616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:43.257653 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:43.257693 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:43.258724 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.259409 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.260461 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.260743 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.260833 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:43.262411 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:43.262414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:43.262417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:43.262439 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:43.264247 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.264766 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.264769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.264771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.264853 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.265156 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.265159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.265161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.265356 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:43.265468 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:43.266154 (+ 686us) server_negotiation.cc:299] Negotiation successful
0928 06:47:43.266278 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:47:43.266076 20948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16219647b37942c5b45f3b595fccc0c5, State: Initialized, Role: FOLLOWER
I20260928 06:47:43.267037 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.256568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55516 (local address 127.7.48.61:36723)
0928 06:47:43.256672 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:43.256675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:43.257407 (+ 732us) server_negotiation.cc:407] Connection header received
0928 06:47:43.257532 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:43.257536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:43.257582 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:43.257656 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:43.258455 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.259148 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.260206 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.260506 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.260590 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:43.263569 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:43.263578 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:43.263583 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:43.263614 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:43.265293 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.265799 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.265803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.265805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.265893 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.266184 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.266187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.266188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.266395 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:43.266512 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:43.266774 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 06:47:43.266883 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:47:43.267571 20948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8335216a7641a89a55faab75ad922c" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36655 } } peers { permanent_uuid: "c4b573c7762c41e5b18139dd2a3b7fad" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 } } peers { permanent_uuid: "16219647b37942c5b45f3b595fccc0c5" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } }
I20260928 06:47:43.267921 20872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } attempt: 3
I20260928 06:47:43.268086 20961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8335216a7641a89a55faab75ad922c" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36655 } } peers { permanent_uuid: "c4b573c7762c41e5b18139dd2a3b7fad" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 } } peers { permanent_uuid: "16219647b37942c5b45f3b595fccc0c5" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } } }
I20260928 06:47:43.268292 20961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:47:43.268633 20797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } attempt: 3
I20260928 06:47:43.268980 20948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 06:47:43 dist-test-slave-twl6 krb5kdc[20716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578062, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.61@KRBTEST.COM for kudu/127.7.48.60@KRBTEST.COM
Sep 28 06:47:43 dist-test-slave-twl6 krb5kdc[20716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578062, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for kudu/127.7.48.60@KRBTEST.COM
W20260928 06:47:43.275264 20973 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:47:43.275385 20973 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:47:43.277215 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.268255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:45337 (local address 127.0.0.1:49372)
0928 06:47:43.268445 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:43.268458 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:43.268523 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:43.268931 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:43.268935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:43.269109 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:47:43.269210 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:43.269216 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.270071 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.270075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:43.270853 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:43.270866 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.271446 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.271450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:43.271534 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:43.272037 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:47:43.272056 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:47:43.273525 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:47:43.275562 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:43.275565 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:43.275567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:43.275818 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:43.276187 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:43.276190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:43.276192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:43.276280 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:43.276706 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:47:43.276709 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:47:43.276800 (+ 91us) client_negotiation.cc:769] Sending connection context
0928 06:47:43.276901 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 06:47:43.277024 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260928 06:47:43.277765 20951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.268655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49372 (local address 127.7.48.60:45337)
0928 06:47:43.268742 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:43.268745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:43.268756 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:47:43.268785 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:43.268788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:43.268821 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:43.268892 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:43.269293 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.269945 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.271019 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.271311 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.271435 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:43.273660 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:43.273664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:43.273666 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:43.273688 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:43.275444 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.275963 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.275967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.275968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.276066 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.276372 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.276377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.276380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.276592 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:43.276694 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:43.277508 (+ 814us) server_negotiation.cc:299] Negotiation successful
0928 06:47:43.277623 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:47:43.273864 20879 master_runner.cc:186] Error getting master registration for 127.7.48.62:36655: OK, instance_id { permanent_uuid: "bb8335216a7641a89a55faab75ad922c" instance_seqno: 1790578062806163 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:47:43.279803 20879 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:47:43.280041 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.268960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:45337 (local address 127.0.0.1:49376)
0928 06:47:43.269074 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:43.269088 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:43.269161 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:43.269751 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:43.269754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:43.269915 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:47:43.270008 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:43.270013 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.271000 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.271003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:43.271717 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:43.271729 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.272402 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.272405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:43.272482 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:43.272877 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:47:43.272893 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:47:43.275312 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:47:43.278326 (+ 3014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:43.278329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:43.278331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:43.278585 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:43.278928 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:43.278931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:43.278933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:43.279022 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:43.279486 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:47:43.279502 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:47:43.279599 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 06:47:43.279703 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 06:47:43.279832 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 06:47:43.280632 20962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:43.269335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49376 (local address 127.7.48.60:45337)
0928 06:47:43.269573 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:43.269579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:43.269589 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:47:43.269614 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:43.269616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:43.269642 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:43.269732 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:43.270114 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.270859 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.271851 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:43.272205 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:43.272300 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:43.275929 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:43.275935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:43.275938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:43.275960 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:43.278202 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.278721 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.278725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.278726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.278823 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:43.279141 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:43.279144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:43.279145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:43.279350 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:43.279438 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:43.280348 (+ 910us) server_negotiation.cc:299] Negotiation successful
0928 06:47:43.280461 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":111,"threads_started":1}
I20260928 06:47:43.280866 20872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4b573c7762c41e5b18139dd2a3b7fad: Bootstrap starting.
I20260928 06:47:43.283350 20797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb8335216a7641a89a55faab75ad922c: Bootstrap starting.
I20260928 06:47:43.283535 20872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4b573c7762c41e5b18139dd2a3b7fad: Neither blocks nor log segments found. Creating new log.
I20260928 06:47:43.284395 20872 log.cc:824] T 00000000000000000000000000000000 P c4b573c7762c41e5b18139dd2a3b7fad: Log is configured to *not* fsync() on all Append() calls
I20260928 06:47:43.286715 20872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4b573c7762c41e5b18139dd2a3b7fad: No bootstrap required, opened a new log
I20260928 06:47:43.287490 20797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb8335216a7641a89a55faab75ad922c: Neither blocks nor log segments found. Creating new log.
I20260928 06:47:43.288321 20797 log.cc:824] T 00000000000000000000000000000000 P bb8335216a7641a89a55faab75ad922c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:47:43.290355 20797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb8335216a7641a89a55faab75ad922c: No bootstrap required, opened a new log
I20260928 06:47:43.290601 20872 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4b573c7762c41e5b18139dd2a3b7fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8335216a7641a89a55faab75ad922c" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36655 } } peers { permanent_uuid: "c4b573c7762c41e5b18139dd2a3b7fad" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 } } peers { permanent_uuid: "16219647b37942c5b45f3b595fccc0c5" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } }
I20260928 06:47:43.290825 20872 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4b573c7762c41e5b18139dd2a3b7fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:47:43.290884 20872 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4b573c7762c41e5b18139dd2a3b7fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4b573c7762c41e5b18139dd2a3b7fad, State: Initialized, Role: FOLLOWER
I20260928 06:47:43.291326 20872 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4b573c7762c41e5b18139dd2a3b7fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8335216a7641a89a55faab75ad922c" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36655 } } peers { permanent_uuid: "c4b573c7762c41e5b18139dd2a3b7fad" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 } } peers { permanent_uuid: "16219647b37942c5b45f3b595fccc0c5" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } }
I20260928 06:47:43.291792 20977 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4b573c7762c41e5b18139dd2a3b7fad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8335216a7641a89a55faab75ad922c" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36655 } } peers { permanent_uuid: "c4b573c7762c41e5b18139dd2a3b7fad" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 } } peers { permanent_uuid: "16219647b37942c5b45f3b595fccc0c5" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } } }
I20260928 06:47:43.292022 20977 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4b573c7762c41e5b18139dd2a3b7fad [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:47:43.292688 20872 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4b573c7762c41e5b18139dd2a3b7fad [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:47:43.296542 20797 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb8335216a7641a89a55faab75ad922c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8335216a7641a89a55faab75ad922c" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36655 } } peers { permanent_uuid: "c4b573c7762c41e5b18139dd2a3b7fad" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 } } peers { permanent_uuid: "16219647b37942c5b45f3b595fccc0c5" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } }
I20260928 06:47:43.296820 20797 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb8335216a7641a89a55faab75ad922c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:47:43.296911 20797 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb8335216a7641a89a55faab75ad922c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb8335216a7641a89a55faab75ad922c, State: Initialized, Role: FOLLOWER
I20260928 06:47:43.297446 20797 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb8335216a7641a89a55faab75ad922c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8335216a7641a89a55faab75ad922c" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36655 } } peers { permanent_uuid: "c4b573c7762c41e5b18139dd2a3b7fad" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 } } peers { permanent_uuid: "16219647b37942c5b45f3b595fccc0c5" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } }
W20260928 06:47:43.298087 20988 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4b573c7762c41e5b18139dd2a3b7fad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:47:43.298166 20988 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:47:43.298872 20797 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb8335216a7641a89a55faab75ad922c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:47:43.296494 20804 master_runner.cc:186] Error getting master registration for 127.7.48.62:36655: OK, instance_id { permanent_uuid: "bb8335216a7641a89a55faab75ad922c" instance_seqno: 1790578062806163 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:47:43.299719 20804 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 06:47:43.302577 20750 tablet.cc:2369] T 00000000000000000000000000000000 P bb8335216a7641a89a55faab75ad922c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:47:43.304273 21001 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb8335216a7641a89a55faab75ad922c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:47:43.304409 21001 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:47:43.304903 20990 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb8335216a7641a89a55faab75ad922c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8335216a7641a89a55faab75ad922c" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36655 } } peers { permanent_uuid: "c4b573c7762c41e5b18139dd2a3b7fad" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 } } peers { permanent_uuid: "16219647b37942c5b45f3b595fccc0c5" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } } }
I20260928 06:47:43.305076 20990 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb8335216a7641a89a55faab75ad922c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:47:43.309093 20729 master_runner.cc:263] No leader master found from master bb8335216a7641a89a55faab75ad922c
I20260928 06:47:43.309154 20729 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:47:43.394878 20961 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:47:43.395107 20961 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8335216a7641a89a55faab75ad922c" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36655 } } peers { permanent_uuid: "c4b573c7762c41e5b18139dd2a3b7fad" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 } } peers { permanent_uuid: "16219647b37942c5b45f3b595fccc0c5" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } }
I20260928 06:47:43.396162 20961 leader_election.cc:288] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb8335216a7641a89a55faab75ad922c (127.7.48.62:36655), c4b573c7762c41e5b18139dd2a3b7fad (127.7.48.61:36723)
I20260928 06:47:43.396999 20846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "16219647b37942c5b45f3b595fccc0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4b573c7762c41e5b18139dd2a3b7fad" is_pre_election: true
I20260928 06:47:43.397303 20846 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c4b573c7762c41e5b18139dd2a3b7fad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16219647b37942c5b45f3b595fccc0c5 in term 0.
I20260928 06:47:43.397861 20898 leader_election.cc:302] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16219647b37942c5b45f3b595fccc0c5, c4b573c7762c41e5b18139dd2a3b7fad; no voters:
I20260928 06:47:43.398670 20961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:47:43.398628 20771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "16219647b37942c5b45f3b595fccc0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8335216a7641a89a55faab75ad922c" is_pre_election: true
I20260928 06:47:43.398758 20961 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:47:43.398801 20961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:47:43.398900 20771 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bb8335216a7641a89a55faab75ad922c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16219647b37942c5b45f3b595fccc0c5 in term 0.
I20260928 06:47:43.400168 20961 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8335216a7641a89a55faab75ad922c" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36655 } } peers { permanent_uuid: "c4b573c7762c41e5b18139dd2a3b7fad" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 } } peers { permanent_uuid: "16219647b37942c5b45f3b595fccc0c5" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } }
I20260928 06:47:43.400483 20961 leader_election.cc:288] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [CANDIDATE]: Term 1 election: Requested vote from peers bb8335216a7641a89a55faab75ad922c (127.7.48.62:36655), c4b573c7762c41e5b18139dd2a3b7fad (127.7.48.61:36723)
I20260928 06:47:43.401014 20846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "16219647b37942c5b45f3b595fccc0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4b573c7762c41e5b18139dd2a3b7fad"
I20260928 06:47:43.401144 20846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4b573c7762c41e5b18139dd2a3b7fad [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:47:43.402369 20846 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c4b573c7762c41e5b18139dd2a3b7fad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16219647b37942c5b45f3b595fccc0c5 in term 1.
I20260928 06:47:43.402719 20898 leader_election.cc:302] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16219647b37942c5b45f3b595fccc0c5, c4b573c7762c41e5b18139dd2a3b7fad; no voters:
I20260928 06:47:43.403648 20771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "16219647b37942c5b45f3b595fccc0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8335216a7641a89a55faab75ad922c"
I20260928 06:47:43.403764 20771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb8335216a7641a89a55faab75ad922c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:47:43.405026 20771 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bb8335216a7641a89a55faab75ad922c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16219647b37942c5b45f3b595fccc0c5 in term 1.
I20260928 06:47:43.405478 20961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:47:43.405756 20961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [term 1 LEADER]: Becoming Leader. State: Replica: 16219647b37942c5b45f3b595fccc0c5, State: Running, Role: LEADER
I20260928 06:47:43.406170 20961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8335216a7641a89a55faab75ad922c" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36655 } } peers { permanent_uuid: "c4b573c7762c41e5b18139dd2a3b7fad" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 } } peers { permanent_uuid: "16219647b37942c5b45f3b595fccc0c5" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } }
I20260928 06:47:43.408134 20961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16219647b37942c5b45f3b595fccc0c5. Latest consensus state: current_term: 1 leader_uuid: "16219647b37942c5b45f3b595fccc0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8335216a7641a89a55faab75ad922c" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36655 } } peers { permanent_uuid: "c4b573c7762c41e5b18139dd2a3b7fad" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 } } peers { permanent_uuid: "16219647b37942c5b45f3b595fccc0c5" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } } }
I20260928 06:47:43.408305 20961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [sys.catalog]: This master's current role is: LEADER
I20260928 06:47:43.412171 21005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:47:43.413129 21005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:47:43.422163 20846 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c4b573c7762c41e5b18139dd2a3b7fad [term 1 FOLLOWER]: Refusing update from remote peer 16219647b37942c5b45f3b595fccc0c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:47:43.422847 21003 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4b573c7762c41e5b18139dd2a3b7fad" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:47:43.427237 20977 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4b573c7762c41e5b18139dd2a3b7fad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16219647b37942c5b45f3b595fccc0c5. Latest consensus state: current_term: 1 leader_uuid: "16219647b37942c5b45f3b595fccc0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8335216a7641a89a55faab75ad922c" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36655 } } peers { permanent_uuid: "c4b573c7762c41e5b18139dd2a3b7fad" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 } } peers { permanent_uuid: "16219647b37942c5b45f3b595fccc0c5" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } } }
I20260928 06:47:43.427350 20977 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4b573c7762c41e5b18139dd2a3b7fad [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:47:43.428360 20771 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bb8335216a7641a89a55faab75ad922c [term 1 FOLLOWER]: Refusing update from remote peer 16219647b37942c5b45f3b595fccc0c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:47:43.428934 21003 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8335216a7641a89a55faab75ad922c" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:47:43.433112 20990 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb8335216a7641a89a55faab75ad922c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16219647b37942c5b45f3b595fccc0c5. Latest consensus state: current_term: 1 leader_uuid: "16219647b37942c5b45f3b595fccc0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8335216a7641a89a55faab75ad922c" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36655 } } peers { permanent_uuid: "c4b573c7762c41e5b18139dd2a3b7fad" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 } } peers { permanent_uuid: "16219647b37942c5b45f3b595fccc0c5" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } } }
I20260928 06:47:43.433236 20990 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb8335216a7641a89a55faab75ad922c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:47:43.437597 21003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16219647b37942c5b45f3b595fccc0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8335216a7641a89a55faab75ad922c" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36655 } } peers { permanent_uuid: "c4b573c7762c41e5b18139dd2a3b7fad" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 } } peers { permanent_uuid: "16219647b37942c5b45f3b595fccc0c5" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } } }
I20260928 06:47:43.437732 21003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [sys.catalog]: This master's current role is: LEADER
I20260928 06:47:43.441128 20990 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb8335216a7641a89a55faab75ad922c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16219647b37942c5b45f3b595fccc0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8335216a7641a89a55faab75ad922c" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36655 } } peers { permanent_uuid: "c4b573c7762c41e5b18139dd2a3b7fad" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 } } peers { permanent_uuid: "16219647b37942c5b45f3b595fccc0c5" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } } }
I20260928 06:47:43.441233 20990 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb8335216a7641a89a55faab75ad922c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:47:43.441900 21006 mvcc.cc:205] Tried to move back new op lower bound from 7334207747755532288 to 7334207747715235840. Current Snapshot: MvccSnapshot[applied={T|T < 7334207747755532288}]
I20260928 06:47:43.444103 21009 mvcc.cc:205] Tried to move back new op lower bound from 7334207747755532288 to 7334207747715235840. Current Snapshot: MvccSnapshot[applied={T|T < 7334207747755532288}]
I20260928 06:47:43.444911 20961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16219647b37942c5b45f3b595fccc0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8335216a7641a89a55faab75ad922c" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36655 } } peers { permanent_uuid: "c4b573c7762c41e5b18139dd2a3b7fad" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 } } peers { permanent_uuid: "16219647b37942c5b45f3b595fccc0c5" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } } }
I20260928 06:47:43.445041 20961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5 [sys.catalog]: This master's current role is: LEADER
I20260928 06:47:43.446098 20977 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4b573c7762c41e5b18139dd2a3b7fad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16219647b37942c5b45f3b595fccc0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8335216a7641a89a55faab75ad922c" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36655 } } peers { permanent_uuid: "c4b573c7762c41e5b18139dd2a3b7fad" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 36723 } } peers { permanent_uuid: "16219647b37942c5b45f3b595fccc0c5" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 45337 } } }
I20260928 06:47:43.446223 20977 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4b573c7762c41e5b18139dd2a3b7fad [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:47:43.448843 21007 mvcc.cc:205] Tried to move back new op lower bound from 7334207747755532288 to 7334207747715235840. Current Snapshot: MvccSnapshot[applied={T|T < 7334207747755532288}]
I20260928 06:47:43.450670 21005 catalog_manager.cc:1347] Generated new cluster ID: 1b9319cc499843759c5b0a000f15d92f
I20260928 06:47:43.450749 21005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:47:43.479291 21005 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:47:43.482254 21005 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:47:43.490151 21005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16219647b37942c5b45f3b595fccc0c5: Generated new TSK 0
I20260928 06:47:43.491591 21005 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:47:44.299525 20988 catalog_manager.cc:1259] Loaded cluster ID: 1b9319cc499843759c5b0a000f15d92f
I20260928 06:47:44.299600 20988 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c4b573c7762c41e5b18139dd2a3b7fad: loading cluster ID for follower catalog manager: success
I20260928 06:47:44.301638 20988 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c4b573c7762c41e5b18139dd2a3b7fad: acquiring CA information for follower catalog manager: success
I20260928 06:47:44.303272 20988 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c4b573c7762c41e5b18139dd2a3b7fad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:47:44.305943 21001 catalog_manager.cc:1259] Loaded cluster ID: 1b9319cc499843759c5b0a000f15d92f
I20260928 06:47:44.306031 21001 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bb8335216a7641a89a55faab75ad922c: loading cluster ID for follower catalog manager: success
I20260928 06:47:44.308130 21001 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bb8335216a7641a89a55faab75ad922c: acquiring CA information for follower catalog manager: success
I20260928 06:47:44.309175 21001 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bb8335216a7641a89a55faab75ad922c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:47:44.549822 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 20729
I20260928 06:47:44.556524 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 20804
W20260928 06:47:44.562256 20898 connection.cc:535] client connection to 127.7.48.61:36723 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 06:47:44.562453 20898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:47:44.562538 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 20879
2026-09-28T06:47:44Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3040 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:47:44 dist-test-slave-twl6 krb5kdc[21021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:47:44 dist-test-slave-twl6 krb5kdc[21021](info): set up 2 sockets
Sep 28 06:47:44 dist-test-slave-twl6 krb5kdc[21021](info): commencing operation
krb5kdc: starting...
W20260928 06:47:45.556442 7360 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:47:45 dist-test-slave-twl6 krb5kdc[21021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:47:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:47:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:47:45.675088 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:39391
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:39391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:47:45.783219 21034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:47:45.783543 21034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:47:45.783612 21034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:47:45.788158 21034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:47:45.788260 21034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:47:45.788291 21034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:47:45.788331 21034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:47:45.788367 21034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:47:45.794072 21034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:39391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:39391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:47:45.795127 21034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:47:45.796211 21034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:47:45.804306 21040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:45.804380 21042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:45.804267 21039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:45.804504 21034 server_base.cc:1034] running on GCE node
I20260928 06:47:45.805218 21034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:47:45.806350 21034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:47:45.807538 21034 hybrid_clock.cc:648] HybridClock initialized: now 1790578065807530 us; error 29 us; skew 500 ppm
Sep 28 06:47:45 dist-test-slave-twl6 krb5kdc[21021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578065, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:45.810321 21034 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:47:45.811223 21034 webserver.cc:460] Webserver started at http://127.7.48.62:46473/ using document root <none> and password file <none>
I20260928 06:47:45.811746 21034 fs_manager.cc:359] Metadata directory not provided
I20260928 06:47:45.811805 21034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:47:45.812036 21034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:47:45.814079 21034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "4c0a5c5cc81d47aeb29050af362eb0f6"
format_stamp: "Formatted at 2026-09-28 06:47:45 on dist-test-slave-twl6"
server_key: "89f880caa10f8bc22c767523d2a7ef57"
server_key_iv: "9a80ef8905742e87e1573d37c1bb1593"
server_key_version: "encryptionkey@0"
I20260928 06:47:45.814599 21034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "4c0a5c5cc81d47aeb29050af362eb0f6"
format_stamp: "Formatted at 2026-09-28 06:47:45 on dist-test-slave-twl6"
server_key: "89f880caa10f8bc22c767523d2a7ef57"
server_key_iv: "9a80ef8905742e87e1573d37c1bb1593"
server_key_version: "encryptionkey@0"
I20260928 06:47:45.818423 21034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 06:47:45.820762 21049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:45.822001 21034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:47:45.822098 21034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "4c0a5c5cc81d47aeb29050af362eb0f6"
format_stamp: "Formatted at 2026-09-28 06:47:45 on dist-test-slave-twl6"
server_key: "89f880caa10f8bc22c767523d2a7ef57"
server_key_iv: "9a80ef8905742e87e1573d37c1bb1593"
server_key_version: "encryptionkey@0"
I20260928 06:47:45.822188 21034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:47:45.834941 21034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:47:45.837659 21034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:47:45.837827 21034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:47:45.844444 21034 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:39391
I20260928 06:47:45.844468 21101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:39391 every 8 connection(s)
I20260928 06:47:45.845362 21034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:47:45.848148 21102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:47:45.851675 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 21034
I20260928 06:47:45.851764 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:47:45.852048 7360 external_mini_cluster.cc:1428] Setting key a3d2aae08b25a1e8065c5f09f88dc57d
I20260928 06:47:45.853722 21102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c0a5c5cc81d47aeb29050af362eb0f6: Bootstrap starting.
I20260928 06:47:45.855841 21102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c0a5c5cc81d47aeb29050af362eb0f6: Neither blocks nor log segments found. Creating new log.
I20260928 06:47:45.856598 21102 log.cc:824] T 00000000000000000000000000000000 P 4c0a5c5cc81d47aeb29050af362eb0f6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:47:45.858795 21102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c0a5c5cc81d47aeb29050af362eb0f6: No bootstrap required, opened a new log
Sep 28 06:47:45 dist-test-slave-twl6 krb5kdc[21021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:47:45.864059 21102 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c0a5c5cc81d47aeb29050af362eb0f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0a5c5cc81d47aeb29050af362eb0f6" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 39391 } }
I20260928 06:47:45.864383 21102 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c0a5c5cc81d47aeb29050af362eb0f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:47:45.864471 21102 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c0a5c5cc81d47aeb29050af362eb0f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c0a5c5cc81d47aeb29050af362eb0f6, State: Initialized, Role: FOLLOWER
I20260928 06:47:45.865002 21102 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c0a5c5cc81d47aeb29050af362eb0f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0a5c5cc81d47aeb29050af362eb0f6" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 39391 } }
I20260928 06:47:45.865170 21102 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c0a5c5cc81d47aeb29050af362eb0f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:47:45.865255 21102 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c0a5c5cc81d47aeb29050af362eb0f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:47:45.865355 21102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c0a5c5cc81d47aeb29050af362eb0f6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:47:45.866494 21102 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c0a5c5cc81d47aeb29050af362eb0f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0a5c5cc81d47aeb29050af362eb0f6" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 39391 } }
I20260928 06:47:45.866854 21102 leader_election.cc:302] T 00000000000000000000000000000000 P 4c0a5c5cc81d47aeb29050af362eb0f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c0a5c5cc81d47aeb29050af362eb0f6; no voters:
I20260928 06:47:45.867167 21102 leader_election.cc:288] T 00000000000000000000000000000000 P 4c0a5c5cc81d47aeb29050af362eb0f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:47:45.867172 21105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:45.853412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42030 (local address 127.7.48.62:39391)
0928 06:47:45.853898 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:45.853914 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:45.853950 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:47:45.854555 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:45.854571 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:45.854900 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:45.855242 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:45.856103 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:45.857251 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:45.858426 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:45.858886 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:45.859057 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:45.862081 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:45.862086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:45.862099 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:45.862126 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:45.863948 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:45.864421 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:45.864427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:45.864433 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:45.864608 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:45.864893 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:45.864895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:45.864897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:45.865418 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:45.865612 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:45.865907 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 06:47:45.866249 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":135,"threads_started":1}
I20260928 06:47:45.867280 21107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c0a5c5cc81d47aeb29050af362eb0f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:47:45.867489 21107 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c0a5c5cc81d47aeb29050af362eb0f6 [term 1 LEADER]: Becoming Leader. State: Replica: 4c0a5c5cc81d47aeb29050af362eb0f6, State: Running, Role: LEADER
I20260928 06:47:45.867843 21107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c0a5c5cc81d47aeb29050af362eb0f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0a5c5cc81d47aeb29050af362eb0f6" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 39391 } }
I20260928 06:47:45.868633 21102 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c0a5c5cc81d47aeb29050af362eb0f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:47:45.869084 21107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0a5c5cc81d47aeb29050af362eb0f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c0a5c5cc81d47aeb29050af362eb0f6. Latest consensus state: current_term: 1 leader_uuid: "4c0a5c5cc81d47aeb29050af362eb0f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0a5c5cc81d47aeb29050af362eb0f6" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 39391 } } }
I20260928 06:47:45.869270 21107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0a5c5cc81d47aeb29050af362eb0f6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:47:45.869798 21107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0a5c5cc81d47aeb29050af362eb0f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c0a5c5cc81d47aeb29050af362eb0f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0a5c5cc81d47aeb29050af362eb0f6" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 39391 } } }
I20260928 06:47:45.869944 21107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0a5c5cc81d47aeb29050af362eb0f6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:47:45.870555 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:47:45.870244 21112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:47:45.873600 21112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:47:45.879087 21112 catalog_manager.cc:1347] Generated new cluster ID: 556abcacc0c141c7a69d125fc20f5be1
I20260928 06:47:45.879158 21112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:47:45.885562 21112 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:47:45.886788 21112 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:47:45.896270 21112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c0a5c5cc81d47aeb29050af362eb0f6: Generated new TSK 0
I20260928 06:47:45.897086 21112 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:47:45.926952 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 21034
2026-09-28T06:47:45Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1363 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:47:45 dist-test-slave-twl6 krb5kdc[21130](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:47:45 dist-test-slave-twl6 krb5kdc[21130](info): set up 2 sockets
Sep 28 06:47:45 dist-test-slave-twl6 krb5kdc[21130](info): commencing operation
krb5kdc: starting...
W20260928 06:47:46.956583 7360 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:47:47 dist-test-slave-twl6 krb5kdc[21130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:47:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:47:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:47:47.080250 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:42645
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:42645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:47:47.193722 21143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:47:47.194078 21143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:47:47.194154 21143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:47:47.198271 21143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:47:47.198382 21143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:47:47.198415 21143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:47:47.198455 21143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:47:47.198484 21143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:47:47.204079 21143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:42645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:42645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:47:47.205571 21143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:47:47.206902 21143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:47:47.216084 21148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:47.216079 21149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:47.216144 21143 server_base.cc:1034] running on GCE node
W20260928 06:47:47.216079 21151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:47.217021 21143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:47:47.218235 21143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:47:47.219453 21143 hybrid_clock.cc:648] HybridClock initialized: now 1790578067219422 us; error 44 us; skew 500 ppm
Sep 28 06:47:47 dist-test-slave-twl6 krb5kdc[21130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578067, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:47.229161 21143 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:47:47.230259 21143 webserver.cc:460] Webserver started at http://127.7.48.62:44277/ using document root <none> and password file <none>
I20260928 06:47:47.230880 21143 fs_manager.cc:359] Metadata directory not provided
I20260928 06:47:47.230973 21143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:47:47.231213 21143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:47:47.233461 21143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "f139cf2a2416490abb75c93f5952fd72"
format_stamp: "Formatted at 2026-09-28 06:47:47 on dist-test-slave-twl6"
server_key: "a2603e8e989e8be342e7c758b288ade8"
server_key_iv: "a7ce030eba781f58374321d293ff897e"
server_key_version: "encryptionkey@0"
I20260928 06:47:47.234081 21143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "f139cf2a2416490abb75c93f5952fd72"
format_stamp: "Formatted at 2026-09-28 06:47:47 on dist-test-slave-twl6"
server_key: "a2603e8e989e8be342e7c758b288ade8"
server_key_iv: "a7ce030eba781f58374321d293ff897e"
server_key_version: "encryptionkey@0"
I20260928 06:47:47.238158 21143 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:47:47.240907 21158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:47.242172 21143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:47:47.242317 21143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "f139cf2a2416490abb75c93f5952fd72"
format_stamp: "Formatted at 2026-09-28 06:47:47 on dist-test-slave-twl6"
server_key: "a2603e8e989e8be342e7c758b288ade8"
server_key_iv: "a7ce030eba781f58374321d293ff897e"
server_key_version: "encryptionkey@0"
I20260928 06:47:47.242443 21143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:47:47.273387 21143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:47:47.276975 21143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:47:47.277209 21143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:47:47.284869 21143 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:42645
I20260928 06:47:47.285187 21210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:42645 every 8 connection(s)
I20260928 06:47:47.286032 21143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:47:47.287252 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 21143
I20260928 06:47:47.287382 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:47:47.287690 7360 external_mini_cluster.cc:1428] Setting key 884a14a4b2b4a1c968cded7298a287c2
I20260928 06:47:47.289569 21211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:47:47.295517 21211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72: Bootstrap starting.
Sep 28 06:47:47 dist-test-slave-twl6 krb5kdc[21130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:47:47.298249 21211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72: Neither blocks nor log segments found. Creating new log.
I20260928 06:47:47.299156 21211 log.cc:824] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72: Log is configured to *not* fsync() on all Append() calls
I20260928 06:47:47.301681 21211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72: No bootstrap required, opened a new log
I20260928 06:47:47.302798 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:47.289315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49834 (local address 127.7.48.62:42645)
0928 06:47:47.289813 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:47.289831 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:47.289869 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:47:47.290542 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:47.290559 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:47.290897 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:47.291262 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:47.292127 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:47.293145 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:47.294052 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:47.294534 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:47.294719 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:47.297675 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:47.297680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:47.297695 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:47.297723 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:47.299675 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:47.300231 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:47.300236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:47.300241 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:47.300349 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:47.300645 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:47.300647 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:47.300648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:47.301059 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:47.301201 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:47.301638 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 06:47:47.302046 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":152,"threads_started":1}
I20260928 06:47:47.305207 21211 raft_consensus.cc:348] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f139cf2a2416490abb75c93f5952fd72" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42645 } }
I20260928 06:47:47.305517 21211 raft_consensus.cc:374] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:47:47.305596 21211 raft_consensus.cc:729] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f139cf2a2416490abb75c93f5952fd72, State: Initialized, Role: FOLLOWER
I20260928 06:47:47.306042 21211 consensus_queue.cc:260] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f139cf2a2416490abb75c93f5952fd72" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42645 } }
I20260928 06:47:47.306198 21211 raft_consensus.cc:388] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:47:47.306296 21211 raft_consensus.cc:482] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:47:47.306416 21211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:47:47.307363 21211 raft_consensus.cc:504] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f139cf2a2416490abb75c93f5952fd72" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42645 } }
I20260928 06:47:47.307710 21211 leader_election.cc:302] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f139cf2a2416490abb75c93f5952fd72; no voters:
I20260928 06:47:47.308072 21211 leader_election.cc:288] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:47:47.308212 21216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:47:47.308481 21216 raft_consensus.cc:686] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [term 1 LEADER]: Becoming Leader. State: Replica: f139cf2a2416490abb75c93f5952fd72, State: Running, Role: LEADER
I20260928 06:47:47.308813 21216 consensus_queue.cc:237] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f139cf2a2416490abb75c93f5952fd72" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42645 } }
I20260928 06:47:47.309468 21211 sys_catalog.cc:564] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:47:47.310794 21217 sys_catalog.cc:455] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f139cf2a2416490abb75c93f5952fd72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f139cf2a2416490abb75c93f5952fd72" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42645 } } }
I20260928 06:47:47.310796 21218 sys_catalog.cc:455] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f139cf2a2416490abb75c93f5952fd72. Latest consensus state: current_term: 1 leader_uuid: "f139cf2a2416490abb75c93f5952fd72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f139cf2a2416490abb75c93f5952fd72" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42645 } } }
I20260928 06:47:47.310956 21218 sys_catalog.cc:458] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [sys.catalog]: This master's current role is: LEADER
I20260928 06:47:47.310956 21217 sys_catalog.cc:458] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [sys.catalog]: This master's current role is: LEADER
I20260928 06:47:47.311805 21225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:47:47.314615 21225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:47:47.320457 21225 catalog_manager.cc:1347] Generated new cluster ID: d638604e7cee4b45b524b2ff27dd0396
I20260928 06:47:47.320549 21225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:47:47.341697 21225 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:47:47.342727 21225 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:47:47.355559 21225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72: Generated new TSK 0
I20260928 06:47:47.356426 21225 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
I20260928 06:47:47.387528 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:42645
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:47:47.507411 21237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:47:47.507752 21237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:47:47.507841 21237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:47:47.512279 21237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:47:47.512377 21237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:47:47.512468 21237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:47:47.518224 21237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:42645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:47:47.519438 21237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:47:47.520646 21237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:47:47.531082 21243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:47.531086 21242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:47.531082 21245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:47.531354 21237 server_base.cc:1034] running on GCE node
I20260928 06:47:47.531973 21237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:47:47.532665 21237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:47:47.533857 21237 hybrid_clock.cc:648] HybridClock initialized: now 1790578067533836 us; error 43 us; skew 500 ppm
Sep 28 06:47:47 dist-test-slave-twl6 krb5kdc[21130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578067, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:47.536669 21237 init.cc:377] Logged in from keytab as kudu/127.7.48.1@KRBTEST.COM (short username kudu)
I20260928 06:47:47.537598 21237 webserver.cc:460] Webserver started at http://127.7.48.1:38095/ using document root <none> and password file <none>
I20260928 06:47:47.538188 21237 fs_manager.cc:359] Metadata directory not provided
I20260928 06:47:47.538280 21237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:47:47.538520 21237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:47:47.540679 21237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "1bf016438a3e49c2828bdb251e4a1cc4"
format_stamp: "Formatted at 2026-09-28 06:47:47 on dist-test-slave-twl6"
server_key: "ac34a05a8e13f170e2e8885403fc9f13"
server_key_iv: "8cfde00221215735cf050cda0ce9c0cc"
server_key_version: "encryptionkey@0"
I20260928 06:47:47.541280 21237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "1bf016438a3e49c2828bdb251e4a1cc4"
format_stamp: "Formatted at 2026-09-28 06:47:47 on dist-test-slave-twl6"
server_key: "ac34a05a8e13f170e2e8885403fc9f13"
server_key_iv: "8cfde00221215735cf050cda0ce9c0cc"
server_key_version: "encryptionkey@0"
I20260928 06:47:47.545269 21237 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 06:47:47.548072 21252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:47.549377 21237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:47:47.549530 21237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "1bf016438a3e49c2828bdb251e4a1cc4"
format_stamp: "Formatted at 2026-09-28 06:47:47 on dist-test-slave-twl6"
server_key: "ac34a05a8e13f170e2e8885403fc9f13"
server_key_iv: "8cfde00221215735cf050cda0ce9c0cc"
server_key_version: "encryptionkey@0"
I20260928 06:47:47.549654 21237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:47:47.569819 21237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:47:47.573166 21237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:47:47.573410 21237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:47:47.574110 21237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:47:47.575109 21237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:47:47.575182 21237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:47.575259 21237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:47:47.575310 21237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:47.584800 21366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:40849 every 8 connection(s)
I20260928 06:47:47.584800 21237 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:40849
I20260928 06:47:47.585850 21237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:47:47.595193 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 21237
I20260928 06:47:47.595358 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
I20260928 06:47:47.595679 7360 external_mini_cluster.cc:1428] Setting key 861e8a70a439db5ac8c2a27e29d6b539
Sep 28 06:47:47 dist-test-slave-twl6 krb5kdc[21130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578067, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:47:47.601209 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:47.587927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:50221 (local address 127.7.48.62:42645)
0928 06:47:47.588090 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:47.588095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:47.589100 (+ 1005us) server_negotiation.cc:407] Connection header received
0928 06:47:47.590446 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:47.590449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:47.590534 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:47.590610 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:47.591941 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:47.592526 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:47.593788 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:47.594020 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:47.594097 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:47.597191 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:47.597195 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:47.597197 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:47.597227 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:47.598887 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:47.599407 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:47.599412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:47.599415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:47.599523 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:47.599858 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:47.599864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:47.599866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:47.600160 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:47.600259 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:47.600961 (+ 702us) server_negotiation.cc:299] Negotiation successful
0928 06:47:47.601078 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 06:47:47.602640 21369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:47.588417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.1:50221)
0928 06:47:47.588918 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:47.588965 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:47.590202 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:47.590805 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:47.590812 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:47.591202 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:47:47.591749 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:47.591765 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:47.592666 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:47.592669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:47.593669 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:47.593679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:47.594165 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:47.594168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:47.594322 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:47.595131 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:47:47.595149 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:47:47.596974 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:47:47.599017 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:47.599022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:47.599037 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:47.599299 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:47.599621 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:47.599623 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:47.599625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:47.599762 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:47.600289 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:47:47.600299 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:47:47.600684 (+ 385us) client_negotiation.cc:769] Sending connection context
0928 06:47:47.600920 (+ 236us) client_negotiation.cc:240] Negotiation successful
0928 06:47:47.601325 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":141,"threads_started":1}
I20260928 06:47:47.604193 21367 heartbeater.cc:347] Connected to a master server at 127.7.48.62:42645
I20260928 06:47:47.604480 21367 heartbeater.cc:464] Registering TS with master...
I20260928 06:47:47.605050 21367 heartbeater.cc:511] Master 127.7.48.62:42645 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.48.2@KRBTEST.COM; defaulting to no policy
I20260928 06:47:47.606783 21175 ts_manager.cc:194] Registered new tserver with Master: 1bf016438a3e49c2828bdb251e4a1cc4 (127.7.48.1:40849)
I20260928 06:47:47.608094 21175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.1@KRBTEST.COM'} at 127.7.48.1:50221
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
I20260928 06:47:47.624296 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:42645
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:47:47.735771 21372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:47:47.736110 21372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:47:47.736158 21372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:47:47.740183 21372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:47:47.740239 21372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:47:47.740330 21372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:47:47.745678 21372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:42645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:47:47.746799 21372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:47:47.747864 21372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:47:47.758369 21378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:47.758369 21380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:47.758386 21377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:47.758630 21372 server_base.cc:1034] running on GCE node
I20260928 06:47:47.759132 21372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:47:47.759742 21372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:47:47.760944 21372 hybrid_clock.cc:648] HybridClock initialized: now 1790578067760923 us; error 31 us; skew 500 ppm
Sep 28 06:47:47 dist-test-slave-twl6 krb5kdc[21130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578067, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:47.763592 21372 init.cc:377] Logged in from keytab as kudu/127.7.48.2@KRBTEST.COM (short username kudu)
I20260928 06:47:47.764506 21372 webserver.cc:460] Webserver started at http://127.7.48.2:42761/ using document root <none> and password file <none>
I20260928 06:47:47.765177 21372 fs_manager.cc:359] Metadata directory not provided
I20260928 06:47:47.765269 21372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:47:47.765486 21372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:47:47.767576 21372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "09b49b0168804bdea2d1e25832aab4e8"
format_stamp: "Formatted at 2026-09-28 06:47:47 on dist-test-slave-twl6"
server_key: "4b48b55d1a375d523203a34b0db045a3"
server_key_iv: "4c2de843581fdf7f7cce91f7e4118026"
server_key_version: "encryptionkey@0"
I20260928 06:47:47.768174 21372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "09b49b0168804bdea2d1e25832aab4e8"
format_stamp: "Formatted at 2026-09-28 06:47:47 on dist-test-slave-twl6"
server_key: "4b48b55d1a375d523203a34b0db045a3"
server_key_iv: "4c2de843581fdf7f7cce91f7e4118026"
server_key_version: "encryptionkey@0"
I20260928 06:47:47.772258 21372 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 06:47:47.774664 21387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:47.775966 21372 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:47:47.776108 21372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "09b49b0168804bdea2d1e25832aab4e8"
format_stamp: "Formatted at 2026-09-28 06:47:47 on dist-test-slave-twl6"
server_key: "4b48b55d1a375d523203a34b0db045a3"
server_key_iv: "4c2de843581fdf7f7cce91f7e4118026"
server_key_version: "encryptionkey@0"
I20260928 06:47:47.776237 21372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:47:47.792052 21372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:47:47.795318 21372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:47:47.795521 21372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:47:47.796340 21372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:47:47.797256 21372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:47:47.797326 21372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:47.797392 21372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:47:47.797421 21372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:47.806721 21372 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:36585
I20260928 06:47:47.806739 21501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:36585 every 8 connection(s)
I20260928 06:47:47.807617 21372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:47:47.810864 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 21372
I20260928 06:47:47.810999 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:47:47.811261 7360 external_mini_cluster.cc:1428] Setting key 61629f77301d777818298961279a6f89
Sep 28 06:47:47 dist-test-slave-twl6 krb5kdc[21130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578067, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.48.3@KRBTEST.COM; defaulting to no policy
I20260928 06:47:47.824244 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:47.809582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:51873 (local address 127.7.48.62:42645)
0928 06:47:47.809716 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:47.809719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:47.810634 (+ 915us) server_negotiation.cc:407] Connection header received
0928 06:47:47.813199 (+ 2565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:47.813202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:47.813246 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:47.813314 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:47.815028 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:47.815575 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:47.817441 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:47.817657 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:47.817738 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:47.820598 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:47.820602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:47.820604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:47.820629 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:47.822100 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:47.822619 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:47.822623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:47.822625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:47.822707 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:47.823073 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:47.823076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:47.823078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:47.823262 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:47.823370 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:47.823912 (+ 542us) server_negotiation.cc:299] Negotiation successful
0928 06:47:47.824049 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.3@KRBTEST.COM" created.
I20260928 06:47:47.825212 21504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:47.810011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.2:51873)
0928 06:47:47.810462 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:47.810504 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:47.812982 (+ 2478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:47.813533 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:47.813545 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:47.814102 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:47:47.814852 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:47.814872 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:47.815710 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:47.815713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:47.817299 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:47.817311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:47.817814 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:47.817818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:47.818012 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:47.818654 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:47:47.818670 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:47:47.820398 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:47:47.822212 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:47.822218 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:47.822235 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:47.822503 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:47.822801 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:47.822805 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:47.822816 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:47.822961 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:47.823367 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:47:47.823374 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:47:47.823632 (+ 258us) client_negotiation.cc:769] Sending connection context
0928 06:47:47.823823 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 06:47:47.824202 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":123,"threads_started":1}
I20260928 06:47:47.826666 21502 heartbeater.cc:347] Connected to a master server at 127.7.48.62:42645
I20260928 06:47:47.826926 21502 heartbeater.cc:464] Registering TS with master...
I20260928 06:47:47.827422 21502 heartbeater.cc:511] Master 127.7.48.62:42645 requested a full tablet report, sending...
I20260928 06:47:47.828526 21175 ts_manager.cc:194] Registered new tserver with Master: 09b49b0168804bdea2d1e25832aab4e8 (127.7.48.2:36585)
I20260928 06:47:47.829109 21175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.2@KRBTEST.COM'} at 127.7.48.2:51873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
I20260928 06:47:47.838815 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:42645
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:47:47.948820 21507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:47:47.949087 21507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:47:47.949131 21507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:47:47.953035 21507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:47:47.953089 21507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:47:47.953146 21507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:47:47.958521 21507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:42645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:47:47.959507 21507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:47:47.960583 21507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:47:47.970788 21513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:47.970788 21515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:47.971112 21507 server_base.cc:1034] running on GCE node
W20260928 06:47:47.971172 21512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:47.971670 21507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:47:47.972295 21507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:47:47.973495 21507 hybrid_clock.cc:648] HybridClock initialized: now 1790578067973470 us; error 26 us; skew 500 ppm
Sep 28 06:47:47 dist-test-slave-twl6 krb5kdc[21130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578067, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:47.976281 21507 init.cc:377] Logged in from keytab as kudu/127.7.48.3@KRBTEST.COM (short username kudu)
I20260928 06:47:47.977171 21507 webserver.cc:460] Webserver started at http://127.7.48.3:45055/ using document root <none> and password file <none>
I20260928 06:47:47.977691 21507 fs_manager.cc:359] Metadata directory not provided
I20260928 06:47:47.977746 21507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:47:47.977905 21507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:47:47.979970 21507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "f0393ad3546d48b6ac2a6a8243a00244"
format_stamp: "Formatted at 2026-09-28 06:47:47 on dist-test-slave-twl6"
server_key: "912d493adc0818cb3ddb8a118acbb522"
server_key_iv: "b041cd7707319e483b9c8edb8f366a5a"
server_key_version: "encryptionkey@0"
I20260928 06:47:47.980535 21507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "f0393ad3546d48b6ac2a6a8243a00244"
format_stamp: "Formatted at 2026-09-28 06:47:47 on dist-test-slave-twl6"
server_key: "912d493adc0818cb3ddb8a118acbb522"
server_key_iv: "b041cd7707319e483b9c8edb8f366a5a"
server_key_version: "encryptionkey@0"
I20260928 06:47:47.984150 21507 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 06:47:47.986573 21522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:47.987751 21507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:47:47.987915 21507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "f0393ad3546d48b6ac2a6a8243a00244"
format_stamp: "Formatted at 2026-09-28 06:47:47 on dist-test-slave-twl6"
server_key: "912d493adc0818cb3ddb8a118acbb522"
server_key_iv: "b041cd7707319e483b9c8edb8f366a5a"
server_key_version: "encryptionkey@0"
I20260928 06:47:47.988034 21507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:47:48.012955 21507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:47:48.016546 21507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:47:48.016759 21507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:47:48.017403 21507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:47:48.018217 21507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:47:48.018283 21507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:48.018350 21507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:47:48.018389 21507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:48.026932 21507 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:35521
I20260928 06:47:48.026954 21635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:35521 every 8 connection(s)
I20260928 06:47:48.027801 21507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:47:48.035324 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 21507
I20260928 06:47:48.035467 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:47:48.035748 7360 external_mini_cluster.cc:1428] Setting key bb076310f62232e117f1a03ba0e19f08
Sep 28 06:47:48 dist-test-slave-twl6 krb5kdc[21130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578067, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:47:48.041414 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:48.029703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:51209 (local address 127.7.48.62:42645)
0928 06:47:48.029865 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:48.029869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:48.030695 (+ 826us) server_negotiation.cc:407] Connection header received
0928 06:47:48.032039 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:48.032043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:48.032103 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:48.032208 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:48.033397 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.033932 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.035115 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.035343 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.035459 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:48.038197 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:48.038199 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:48.038201 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:48.038226 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:48.039455 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:48.039982 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:48.039985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:48.039988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:48.040100 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:48.040463 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:48.040465 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:48.040466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:48.040648 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:48.040721 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:48.041179 (+ 458us) server_negotiation.cc:299] Negotiation successful
0928 06:47:48.041295 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 06:47:48.042299 21638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:48.030068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.3:51209)
0928 06:47:48.030548 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:48.030591 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:48.031780 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:48.032384 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:48.032391 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:48.032757 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:47:48.033245 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:48.033258 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.034077 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.034081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:48.034951 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:48.034959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.035510 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.035514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:48.035687 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:48.036408 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:47:48.036430 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:47:48.038029 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:47:48.039555 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:48.039559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:48.039572 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:48.039835 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:48.040201 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:48.040204 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:48.040206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:48.040374 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:48.040733 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:47:48.040738 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:47:48.040980 (+ 242us) client_negotiation.cc:769] Sending connection context
0928 06:47:48.041123 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 06:47:48.041397 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":133,"threads_started":1}
I20260928 06:47:48.043401 21636 heartbeater.cc:347] Connected to a master server at 127.7.48.62:42645
I20260928 06:47:48.043658 21636 heartbeater.cc:464] Registering TS with master...
I20260928 06:47:48.044183 21636 heartbeater.cc:511] Master 127.7.48.62:42645 requested a full tablet report, sending...
I20260928 06:47:48.045182 21175 ts_manager.cc:194] Registered new tserver with Master: f0393ad3546d48b6ac2a6a8243a00244 (127.7.48.3:35521)
I20260928 06:47:48.045758 21175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.3@KRBTEST.COM'} at 127.7.48.3:51209
I20260928 06:47:48.049681 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:47:48.059266 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:48.051541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49838 (local address 127.7.48.62:42645)
0928 06:47:48.051690 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:48.051695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:48.051890 (+ 195us) server_negotiation.cc:407] Connection header received
0928 06:47:48.052059 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:48.052062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:48.052110 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:48.052193 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:48.052882 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.053704 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.054575 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.054794 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.054901 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:48.056302 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:48.056305 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:48.056308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:48.056336 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:48.057593 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:48.058026 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:48.058030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:48.058032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:48.058114 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:48.058395 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:48.058399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:48.058401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:48.058623 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:48.058765 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:48.058992 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 06:47:48.059104 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 06:47:48.061966 21175 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:47:48.064927 21175 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:47:48.079715 21647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:48.073296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:36585 (local address 127.0.0.1:39062)
0928 06:47:48.073711 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:48.073783 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:48.073918 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:48.075853 (+ 1935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:48.075855 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:48.075871 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:47:48.075991 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:48.075998 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.077146 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.077151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:48.078176 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:48.078187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.079090 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.079094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:48.079219 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:48.079262 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 06:47:48.079421 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 06:47:48.079542 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":55,"thread_start_us":108,"threads_started":1}
I20260928 06:47:48.080188 21652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:48.074439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52278 (local address 127.7.48.3:35521)
0928 06:47:48.074964 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:48.074973 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:48.074989 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:47:48.075048 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:48.075052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:48.075275 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:48.075432 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:47:48.075687 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.076793 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.078788 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.079642 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.079744 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:48.079901 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:48.079972 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 06:47:48.080062 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":142,"threads_started":1}
I20260928 06:47:48.080610 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:48.074221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39062 (local address 127.7.48.2:36585)
0928 06:47:48.075425 (+ 1204us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:48.075437 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:48.075456 (+ 19us) server_negotiation.cc:407] Connection header received
0928 06:47:48.075525 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:48.075530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:48.075712 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:48.075856 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:47:48.076100 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.076803 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.078308 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.078965 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.079079 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:48.079226 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:48.080367 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0928 06:47:48.080470 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1083,"thread_start_us":152,"threads_started":1}
I20260928 06:47:48.080668 21649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:48.073402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55738 (local address 127.7.48.1:40849)
0928 06:47:48.074543 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:48.074563 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:48.074589 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:47:48.074685 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:48.074693 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:48.074989 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:48.075136 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:47:48.075514 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.076789 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.078806 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.079636 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.079753 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:48.079943 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:48.080277 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 06:47:48.080392 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1030,"thread_start_us":105,"threads_started":1}
I20260928 06:47:48.080965 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:48.074014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:35521 (local address 127.0.0.1:52278)
0928 06:47:48.074675 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:48.074688 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:48.074769 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:48.075428 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:48.075433 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:48.075462 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:47:48.075566 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:48.075572 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.076971 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.076974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:48.078653 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:48.078664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.079743 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.079746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:48.079828 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:48.079850 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 06:47:48.079936 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:47:48.080006 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":150,"thread_start_us":103,"threads_started":1}
I20260928 06:47:48.081022 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:48.073294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:40849 (local address 127.0.0.1:55738)
0928 06:47:48.074168 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:48.074181 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:48.074262 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:48.075172 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:48.075178 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:48.075203 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:47:48.075372 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:48.075383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.076971 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.076974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:48.078653 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:48.078664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.080000 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.080003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:48.080104 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:48.080128 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 06:47:48.080829 (+ 701us) client_negotiation.cc:240] Negotiation successful
0928 06:47:48.080903 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"mutex_wait_us":193,"spinlock_wait_cycles":7040,"thread_start_us":103,"threads_started":1}
I20260928 06:47:48.083241 21570 tablet_service.cc:1467] Processing CreateTablet for tablet 3a0f55d777ae4c0b9f75586dbf1c140a (DEFAULT_TABLE table=test-table [id=be0d4e97918843699c24a2d04b18c814]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:47:48.083177 21435 tablet_service.cc:1467] Processing CreateTablet for tablet 3a0f55d777ae4c0b9f75586dbf1c140a (DEFAULT_TABLE table=test-table [id=be0d4e97918843699c24a2d04b18c814]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:47:48.084267 21570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a0f55d777ae4c0b9f75586dbf1c140a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:47:48.084399 21435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a0f55d777ae4c0b9f75586dbf1c140a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:47:48.085660 21301 tablet_service.cc:1467] Processing CreateTablet for tablet 3a0f55d777ae4c0b9f75586dbf1c140a (DEFAULT_TABLE table=test-table [id=be0d4e97918843699c24a2d04b18c814]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:47:48.086663 21301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a0f55d777ae4c0b9f75586dbf1c140a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:47:48.090164 21653 tablet_bootstrap.cc:492] T 3a0f55d777ae4c0b9f75586dbf1c140a P f0393ad3546d48b6ac2a6a8243a00244: Bootstrap starting.
I20260928 06:47:48.092759 21653 tablet_bootstrap.cc:654] T 3a0f55d777ae4c0b9f75586dbf1c140a P f0393ad3546d48b6ac2a6a8243a00244: Neither blocks nor log segments found. Creating new log.
I20260928 06:47:48.093462 21655 tablet_bootstrap.cc:492] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4: Bootstrap starting.
I20260928 06:47:48.093928 21653 log.cc:824] T 3a0f55d777ae4c0b9f75586dbf1c140a P f0393ad3546d48b6ac2a6a8243a00244: Log is configured to *not* fsync() on all Append() calls
I20260928 06:47:48.094061 21654 tablet_bootstrap.cc:492] T 3a0f55d777ae4c0b9f75586dbf1c140a P 09b49b0168804bdea2d1e25832aab4e8: Bootstrap starting.
I20260928 06:47:48.095669 21655 tablet_bootstrap.cc:654] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4: Neither blocks nor log segments found. Creating new log.
I20260928 06:47:48.095686 21653 tablet_bootstrap.cc:492] T 3a0f55d777ae4c0b9f75586dbf1c140a P f0393ad3546d48b6ac2a6a8243a00244: No bootstrap required, opened a new log
I20260928 06:47:48.095872 21653 ts_tablet_manager.cc:1397] T 3a0f55d777ae4c0b9f75586dbf1c140a P f0393ad3546d48b6ac2a6a8243a00244: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 06:47:48.096613 21654 tablet_bootstrap.cc:654] T 3a0f55d777ae4c0b9f75586dbf1c140a P 09b49b0168804bdea2d1e25832aab4e8: Neither blocks nor log segments found. Creating new log.
I20260928 06:47:48.097055 21655 log.cc:824] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4: Log is configured to *not* fsync() on all Append() calls
I20260928 06:47:48.097430 21654 log.cc:824] T 3a0f55d777ae4c0b9f75586dbf1c140a P 09b49b0168804bdea2d1e25832aab4e8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:47:48.098861 21655 tablet_bootstrap.cc:492] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4: No bootstrap required, opened a new log
I20260928 06:47:48.099078 21655 ts_tablet_manager.cc:1397] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260928 06:47:48.099323 21654 tablet_bootstrap.cc:492] T 3a0f55d777ae4c0b9f75586dbf1c140a P 09b49b0168804bdea2d1e25832aab4e8: No bootstrap required, opened a new log
I20260928 06:47:48.099504 21654 ts_tablet_manager.cc:1397] T 3a0f55d777ae4c0b9f75586dbf1c140a P 09b49b0168804bdea2d1e25832aab4e8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 06:47:48.099391 21653 raft_consensus.cc:348] T 3a0f55d777ae4c0b9f75586dbf1c140a P f0393ad3546d48b6ac2a6a8243a00244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } } peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } }
I20260928 06:47:48.099678 21653 raft_consensus.cc:374] T 3a0f55d777ae4c0b9f75586dbf1c140a P f0393ad3546d48b6ac2a6a8243a00244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:47:48.099776 21653 raft_consensus.cc:729] T 3a0f55d777ae4c0b9f75586dbf1c140a P f0393ad3546d48b6ac2a6a8243a00244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0393ad3546d48b6ac2a6a8243a00244, State: Initialized, Role: FOLLOWER
I20260928 06:47:48.100235 21653 consensus_queue.cc:260] T 3a0f55d777ae4c0b9f75586dbf1c140a P f0393ad3546d48b6ac2a6a8243a00244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } } peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } }
I20260928 06:47:48.100942 21636 heartbeater.cc:503] Master 127.7.48.62:42645 was elected leader, sending a full tablet report...
I20260928 06:47:48.101224 21653 ts_tablet_manager.cc:1428] T 3a0f55d777ae4c0b9f75586dbf1c140a P f0393ad3546d48b6ac2a6a8243a00244: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 06:47:48.102703 21655 raft_consensus.cc:348] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } } peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } }
I20260928 06:47:48.102926 21655 raft_consensus.cc:374] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:47:48.103017 21655 raft_consensus.cc:729] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bf016438a3e49c2828bdb251e4a1cc4, State: Initialized, Role: FOLLOWER
I20260928 06:47:48.103067 21654 raft_consensus.cc:348] T 3a0f55d777ae4c0b9f75586dbf1c140a P 09b49b0168804bdea2d1e25832aab4e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } } peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } }
I20260928 06:47:48.103365 21654 raft_consensus.cc:374] T 3a0f55d777ae4c0b9f75586dbf1c140a P 09b49b0168804bdea2d1e25832aab4e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:47:48.103422 21654 raft_consensus.cc:729] T 3a0f55d777ae4c0b9f75586dbf1c140a P 09b49b0168804bdea2d1e25832aab4e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09b49b0168804bdea2d1e25832aab4e8, State: Initialized, Role: FOLLOWER
I20260928 06:47:48.103492 21655 consensus_queue.cc:260] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } } peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } }
I20260928 06:47:48.103858 21654 consensus_queue.cc:260] T 3a0f55d777ae4c0b9f75586dbf1c140a P 09b49b0168804bdea2d1e25832aab4e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } } peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } }
I20260928 06:47:48.104257 21367 heartbeater.cc:503] Master 127.7.48.62:42645 was elected leader, sending a full tablet report...
I20260928 06:47:48.104597 21655 ts_tablet_manager.cc:1428] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 06:47:48.105244 21654 ts_tablet_manager.cc:1428] T 3a0f55d777ae4c0b9f75586dbf1c140a P 09b49b0168804bdea2d1e25832aab4e8: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260928 06:47:48.105676 21502 heartbeater.cc:503] Master 127.7.48.62:42645 was elected leader, sending a full tablet report...
I20260928 06:47:48.208072 21660 raft_consensus.cc:482] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:47:48.208355 21660 raft_consensus.cc:504] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } } peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } }
I20260928 06:47:48.209604 21660 leader_election.cc:288] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09b49b0168804bdea2d1e25832aab4e8 (127.7.48.2:36585), f0393ad3546d48b6ac2a6a8243a00244 (127.7.48.3:35521)
I20260928 06:47:48.213719 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:48.209942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:43137 (local address 127.7.48.2:36585)
0928 06:47:48.210086 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:48.210093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:48.210357 (+ 264us) server_negotiation.cc:407] Connection header received
0928 06:47:48.210495 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:48.210503 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:48.210591 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:48.210673 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:47:48.210981 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.211488 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.212729 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.213117 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.213242 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:48.213338 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:48.213529 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 06:47:48.213598 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 06:47:48.213737 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:48.209830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:36585 (local address 127.7.48.1:43137)
0928 06:47:48.210247 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:48.210259 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:48.210397 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:48.210697 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:48.210700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:48.210715 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:47:48.210863 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:48.210869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.211598 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.211603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:48.212596 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:48.212606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.213248 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.213251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:48.213376 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:48.213407 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 06:47:48.213512 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 06:47:48.213584 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":35,"thread_start_us":149,"threads_started":1}
I20260928 06:47:48.213764 21652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:48.209994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:55627 (local address 127.7.48.3:35521)
0928 06:47:48.210118 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:48.210123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:48.210372 (+ 249us) server_negotiation.cc:407] Connection header received
0928 06:47:48.210530 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:48.210551 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:48.210655 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:48.210731 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:47:48.211073 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.211568 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.212716 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.213138 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.213254 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:48.213317 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:48.213574 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 06:47:48.213650 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:47:48.214169 21662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:48.209878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:35521 (local address 127.7.48.1:55627)
0928 06:47:48.210277 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:48.210288 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:48.210411 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:48.210740 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:48.210743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:48.210760 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:47:48.210956 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:48.210962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.211676 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.211681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:48.212595 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:48.212605 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.213259 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.213262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:48.213407 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:48.213450 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 06:47:48.213932 (+ 482us) client_negotiation.cc:240] Negotiation successful
0928 06:47:48.214016 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":56,"thread_start_us":114,"threads_started":1}
I20260928 06:47:48.214455 21456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0f55d777ae4c0b9f75586dbf1c140a" candidate_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09b49b0168804bdea2d1e25832aab4e8" is_pre_election: true
I20260928 06:47:48.214787 21456 raft_consensus.cc:2454] T 3a0f55d777ae4c0b9f75586dbf1c140a P 09b49b0168804bdea2d1e25832aab4e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bf016438a3e49c2828bdb251e4a1cc4 in term 0.
I20260928 06:47:48.214818 21590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0f55d777ae4c0b9f75586dbf1c140a" candidate_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0393ad3546d48b6ac2a6a8243a00244" is_pre_election: true
I20260928 06:47:48.215098 21590 raft_consensus.cc:2454] T 3a0f55d777ae4c0b9f75586dbf1c140a P f0393ad3546d48b6ac2a6a8243a00244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bf016438a3e49c2828bdb251e4a1cc4 in term 0.
I20260928 06:47:48.215381 21255 leader_election.cc:302] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09b49b0168804bdea2d1e25832aab4e8, 1bf016438a3e49c2828bdb251e4a1cc4; no voters:
I20260928 06:47:48.215677 21660 raft_consensus.cc:2781] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:47:48.215762 21660 raft_consensus.cc:482] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:47:48.215793 21660 raft_consensus.cc:3037] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:47:48.216892 21660 raft_consensus.cc:504] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } } peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } }
I20260928 06:47:48.217263 21660 leader_election.cc:288] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 [CANDIDATE]: Term 1 election: Requested vote from peers 09b49b0168804bdea2d1e25832aab4e8 (127.7.48.2:36585), f0393ad3546d48b6ac2a6a8243a00244 (127.7.48.3:35521)
I20260928 06:47:48.217598 21456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0f55d777ae4c0b9f75586dbf1c140a" candidate_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09b49b0168804bdea2d1e25832aab4e8"
I20260928 06:47:48.217594 21590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0f55d777ae4c0b9f75586dbf1c140a" candidate_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0393ad3546d48b6ac2a6a8243a00244"
I20260928 06:47:48.217703 21456 raft_consensus.cc:3037] T 3a0f55d777ae4c0b9f75586dbf1c140a P 09b49b0168804bdea2d1e25832aab4e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:47:48.217728 21590 raft_consensus.cc:3037] T 3a0f55d777ae4c0b9f75586dbf1c140a P f0393ad3546d48b6ac2a6a8243a00244 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:47:48.218740 21590 raft_consensus.cc:2454] T 3a0f55d777ae4c0b9f75586dbf1c140a P f0393ad3546d48b6ac2a6a8243a00244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bf016438a3e49c2828bdb251e4a1cc4 in term 1.
I20260928 06:47:48.218750 21456 raft_consensus.cc:2454] T 3a0f55d777ae4c0b9f75586dbf1c140a P 09b49b0168804bdea2d1e25832aab4e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bf016438a3e49c2828bdb251e4a1cc4 in term 1.
I20260928 06:47:48.219010 21254 leader_election.cc:302] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bf016438a3e49c2828bdb251e4a1cc4, f0393ad3546d48b6ac2a6a8243a00244; no voters:
I20260928 06:47:48.219206 21660 raft_consensus.cc:2781] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:47:48.219408 21660 raft_consensus.cc:686] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 LEADER]: Becoming Leader. State: Replica: 1bf016438a3e49c2828bdb251e4a1cc4, State: Running, Role: LEADER
I20260928 06:47:48.219697 21660 consensus_queue.cc:237] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } } peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } }
I20260928 06:47:48.222813 21174 catalog_manager.cc:5495] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bf016438a3e49c2828bdb251e4a1cc4 (127.7.48.1). New cstate: current_term: 1 leader_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } health_report { overall_health: UNKNOWN } } }
I20260928 06:47:48.234055 21649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:48.230495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55748 (local address 127.7.48.1:40849)
0928 06:47:48.230621 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:48.230627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:48.230666 (+ 39us) server_negotiation.cc:407] Connection header received
0928 06:47:48.230833 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:48.230836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:48.230900 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:48.231000 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:47:48.231271 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.231788 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.232619 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.232846 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.232949 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:48.233267 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:47:48.233686 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:47:48.233824 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:48.233859 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 06:47:48.233915 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:47:48.239925 21456 raft_consensus.cc:1261] T 3a0f55d777ae4c0b9f75586dbf1c140a P 09b49b0168804bdea2d1e25832aab4e8 [term 1 FOLLOWER]: Refusing update from remote peer 1bf016438a3e49c2828bdb251e4a1cc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:47:48.239871 21590 raft_consensus.cc:1261] T 3a0f55d777ae4c0b9f75586dbf1c140a P f0393ad3546d48b6ac2a6a8243a00244 [term 1 FOLLOWER]: Refusing update from remote peer 1bf016438a3e49c2828bdb251e4a1cc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:47:48.240713 21664 consensus_queue.cc:1035] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:47:48.240904 21660 consensus_queue.cc:1035] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:47:48.248610 21668 mvcc.cc:205] Tried to move back new op lower bound from 7334207767503052800 to 7334207767430443008. Current Snapshot: MvccSnapshot[applied={T|T < 7334207767503052800}]
I20260928 06:47:48.249830 21669 mvcc.cc:205] Tried to move back new op lower bound from 7334207767503052800 to 7334207767430443008. Current Snapshot: MvccSnapshot[applied={T|T < 7334207767503052800}]
I20260928 06:47:48.257118 21175 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:47:48.257277 21175 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:47:48.259625 21175 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72: Sending DeleteTablet for 3 replicas of tablet 3a0f55d777ae4c0b9f75586dbf1c140a
I20260928 06:47:48.260440 21301 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a0f55d777ae4c0b9f75586dbf1c140a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:47:48 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:55738
I20260928 06:47:48.260526 21435 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a0f55d777ae4c0b9f75586dbf1c140a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:47:48 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:39062
I20260928 06:47:48.260684 21570 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a0f55d777ae4c0b9f75586dbf1c140a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:47:48 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:52278
I20260928 06:47:48.260993 21677 tablet_replica.cc:330] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4: stopping tablet replica
I20260928 06:47:48.261281 21678 tablet_replica.cc:330] T 3a0f55d777ae4c0b9f75586dbf1c140a P 09b49b0168804bdea2d1e25832aab4e8: stopping tablet replica
I20260928 06:47:48.261368 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 21237
I20260928 06:47:48.261359 21677 raft_consensus.cc:2229] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:47:48.261729 21678 raft_consensus.cc:2229] T 3a0f55d777ae4c0b9f75586dbf1c140a P 09b49b0168804bdea2d1e25832aab4e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:47:48.262195 21678 raft_consensus.cc:2258] T 3a0f55d777ae4c0b9f75586dbf1c140a P 09b49b0168804bdea2d1e25832aab4e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:47:48.262734 21679 tablet_replica.cc:330] T 3a0f55d777ae4c0b9f75586dbf1c140a P f0393ad3546d48b6ac2a6a8243a00244: stopping tablet replica
I20260928 06:47:48.263069 21679 raft_consensus.cc:2229] T 3a0f55d777ae4c0b9f75586dbf1c140a P f0393ad3546d48b6ac2a6a8243a00244 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:47:48.263381 21679 raft_consensus.cc:2258] T 3a0f55d777ae4c0b9f75586dbf1c140a P f0393ad3546d48b6ac2a6a8243a00244 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:47:48.264053 21678 ts_tablet_manager.cc:1905] T 3a0f55d777ae4c0b9f75586dbf1c140a P 09b49b0168804bdea2d1e25832aab4e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:47:48.264916 21679 ts_tablet_manager.cc:1905] T 3a0f55d777ae4c0b9f75586dbf1c140a P f0393ad3546d48b6ac2a6a8243a00244: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:47:48.266516 21678 ts_tablet_manager.cc:1918] T 3a0f55d777ae4c0b9f75586dbf1c140a P 09b49b0168804bdea2d1e25832aab4e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:47:48.266621 21678 log.cc:1196] T 3a0f55d777ae4c0b9f75586dbf1c140a P 09b49b0168804bdea2d1e25832aab4e8: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/3a0f55d777ae4c0b9f75586dbf1c140a
I20260928 06:47:48.266923 21678 ts_tablet_manager.cc:1939] T 3a0f55d777ae4c0b9f75586dbf1c140a P 09b49b0168804bdea2d1e25832aab4e8: Deleting consensus metadata
I20260928 06:47:48.268246 21161 catalog_manager.cc:4841] TS 09b49b0168804bdea2d1e25832aab4e8 (127.7.48.2:36585): tablet 3a0f55d777ae4c0b9f75586dbf1c140a (table test-table [id=be0d4e97918843699c24a2d04b18c814]) successfully deleted
I20260928 06:47:48.268329 21679 ts_tablet_manager.cc:1918] T 3a0f55d777ae4c0b9f75586dbf1c140a P f0393ad3546d48b6ac2a6a8243a00244: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:47:48.268435 21679 log.cc:1196] T 3a0f55d777ae4c0b9f75586dbf1c140a P f0393ad3546d48b6ac2a6a8243a00244: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/3a0f55d777ae4c0b9f75586dbf1c140a
I20260928 06:47:48.268823 21679 ts_tablet_manager.cc:1939] T 3a0f55d777ae4c0b9f75586dbf1c140a P f0393ad3546d48b6ac2a6a8243a00244: Deleting consensus metadata
W20260928 06:47:48.269712 21162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.48.1:40849 (error 108)
I20260928 06:47:48.270000 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 21372
I20260928 06:47:48.271431 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:48.271131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:40849 (local address 127.0.0.1:55756)
0928 06:47:48.271266 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:48.271344 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.1:40849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 06:47:48.271658 21162 catalog_manager.cc:4584] TS 1bf016438a3e49c2828bdb251e4a1cc4 (127.7.48.1:40849): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3a0f55d777ae4c0b9f75586dbf1c140a: Network error: Client connection negotiation failed: client connection to 127.7.48.1:40849: connect: Connection refused (error 111)
I20260928 06:47:48.271942 21160 catalog_manager.cc:4841] TS f0393ad3546d48b6ac2a6a8243a00244 (127.7.48.3:35521): tablet 3a0f55d777ae4c0b9f75586dbf1c140a (table test-table [id=be0d4e97918843699c24a2d04b18c814]) successfully deleted
I20260928 06:47:48.276418 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 21507
I20260928 06:47:48.282282 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 21143
I20260928 06:47:48.288877 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:42645
--webserver_interface=127.7.48.62
--webserver_port=44277
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:42645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:47:48.398518 21681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:47:48.398835 21681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:47:48.398900 21681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:47:48.402714 21681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:47:48.402807 21681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:47:48.402844 21681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:47:48.402886 21681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:47:48.402936 21681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:47:48.408263 21681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:42645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:42645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=44277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:47:48.409399 21681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:47:48.410454 21681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:47:48.423403 21687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:48.423403 21688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:48.423403 21690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:48.423862 21681 server_base.cc:1034] running on GCE node
I20260928 06:47:48.424453 21681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:47:48.425673 21681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:47:48.426823 21681 hybrid_clock.cc:648] HybridClock initialized: now 1790578068426789 us; error 50 us; skew 500 ppm
Sep 28 06:47:48 dist-test-slave-twl6 krb5kdc[21130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578068, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:48.429510 21681 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:47:48.430426 21681 webserver.cc:460] Webserver started at http://127.7.48.62:44277/ using document root <none> and password file <none>
I20260928 06:47:48.430998 21681 fs_manager.cc:359] Metadata directory not provided
I20260928 06:47:48.431109 21681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:47:48.435164 21681 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:47:48.437427 21697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:48.438686 21681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:47:48.438832 21681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "f139cf2a2416490abb75c93f5952fd72"
format_stamp: "Formatted at 2026-09-28 06:47:47 on dist-test-slave-twl6"
server_key: "a2603e8e989e8be342e7c758b288ade8"
server_key_iv: "a7ce030eba781f58374321d293ff897e"
server_key_version: "encryptionkey@0"
I20260928 06:47:48.439456 21681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:47:48.450579 21681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:47:48.453562 21681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:47:48.453770 21681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:47:48.460783 21681 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:42645
I20260928 06:47:48.461135 21749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:42645 every 8 connection(s)
I20260928 06:47:48.461961 21681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:47:48.465756 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 21681
I20260928 06:47:48.466279 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:40849
--local_ip_for_outbound_sockets=127.7.48.1
--tserver_master_addrs=127.7.48.62:42645
--webserver_port=38095
--webserver_interface=127.7.48.1
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:47:48.467171 21750 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:47:48.469326 21750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72: Bootstrap starting.
I20260928 06:47:48.486814 21750 log.cc:824] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72: Log is configured to *not* fsync() on all Append() calls
I20260928 06:47:48.495414 21750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:47:48.495827 21750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72: Bootstrap complete.
I20260928 06:47:48.499650 21750 raft_consensus.cc:348] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f139cf2a2416490abb75c93f5952fd72" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42645 } }
I20260928 06:47:48.500205 21750 raft_consensus.cc:729] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f139cf2a2416490abb75c93f5952fd72, State: Initialized, Role: FOLLOWER
I20260928 06:47:48.500742 21750 consensus_queue.cc:260] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f139cf2a2416490abb75c93f5952fd72" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42645 } }
I20260928 06:47:48.500890 21750 raft_consensus.cc:388] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:47:48.500988 21750 raft_consensus.cc:482] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:47:48.501123 21750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:47:48.503018 21750 raft_consensus.cc:504] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f139cf2a2416490abb75c93f5952fd72" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42645 } }
I20260928 06:47:48.503360 21750 leader_election.cc:302] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f139cf2a2416490abb75c93f5952fd72; no voters:
I20260928 06:47:48.503719 21750 leader_election.cc:288] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:47:48.503808 21754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:47:48.504046 21754 raft_consensus.cc:686] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [term 2 LEADER]: Becoming Leader. State: Replica: f139cf2a2416490abb75c93f5952fd72, State: Running, Role: LEADER
I20260928 06:47:48.504335 21754 consensus_queue.cc:237] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f139cf2a2416490abb75c93f5952fd72" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42645 } }
I20260928 06:47:48.504977 21750 sys_catalog.cc:564] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:47:48.505270 21754 sys_catalog.cc:455] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f139cf2a2416490abb75c93f5952fd72. Latest consensus state: current_term: 2 leader_uuid: "f139cf2a2416490abb75c93f5952fd72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f139cf2a2416490abb75c93f5952fd72" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42645 } } }
I20260928 06:47:48.505391 21754 sys_catalog.cc:458] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [sys.catalog]: This master's current role is: LEADER
I20260928 06:47:48.505774 21755 sys_catalog.cc:455] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f139cf2a2416490abb75c93f5952fd72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f139cf2a2416490abb75c93f5952fd72" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42645 } } }
I20260928 06:47:48.505934 21755 sys_catalog.cc:458] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [sys.catalog]: This master's current role is: LEADER
I20260928 06:47:48.507023 21758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:47:48.512117 21758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:47:48.512681 21758 catalog_manager.cc:1259] Loaded cluster ID: d638604e7cee4b45b524b2ff27dd0396
I20260928 06:47:48.512739 21758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:47:48.514061 21758 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 06:47:48.518399 21758 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 06:47:48.525154 21758 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:47:48.525965 21758 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72: Loaded TSK: 0
I20260928 06:47:48.527189 21758 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:47:48.582309 21752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:47:48.582583 21752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:47:48.582660 21752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:47:48.586594 21752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:47:48.586689 21752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:47:48.586773 21752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:47:48.591666 21752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:40849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=38095
--tserver_master_addrs=127.7.48.62:42645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:47:48.592705 21752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:47:48.593695 21752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:47:48.602711 21777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:48.602718 21780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:48.602718 21778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:48.603654 21752 server_base.cc:1034] running on GCE node
I20260928 06:47:48.604126 21752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:47:48.604792 21752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:47:48.605998 21752 hybrid_clock.cc:648] HybridClock initialized: now 1790578068605947 us; error 38 us; skew 500 ppm
Sep 28 06:47:48 dist-test-slave-twl6 krb5kdc[21130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578068, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:48.609509 21752 init.cc:377] Logged in from keytab as kudu/127.7.48.1@KRBTEST.COM (short username kudu)
I20260928 06:47:48.610607 21752 webserver.cc:460] Webserver started at http://127.7.48.1:38095/ using document root <none> and password file <none>
I20260928 06:47:48.611299 21752 fs_manager.cc:359] Metadata directory not provided
I20260928 06:47:48.611403 21752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:47:48.616637 21752 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:47:48.619323 21787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:48.620736 21752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:47:48.620865 21752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "1bf016438a3e49c2828bdb251e4a1cc4"
format_stamp: "Formatted at 2026-09-28 06:47:47 on dist-test-slave-twl6"
server_key: "ac34a05a8e13f170e2e8885403fc9f13"
server_key_iv: "8cfde00221215735cf050cda0ce9c0cc"
server_key_version: "encryptionkey@0"
I20260928 06:47:48.621403 21752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:47:48.631381 21752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:47:48.634595 21752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:47:48.634801 21752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:47:48.635603 21752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:47:48.636881 21794 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:47:48.642241 21752 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:47:48.642328 21752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260928 06:47:48.642405 21752 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:47:48.644209 21752 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:47:48.644271 21752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260928 06:47:48.644583 21794 tablet_bootstrap.cc:492] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4: Bootstrap starting.
I20260928 06:47:48.654997 21752 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:40849
I20260928 06:47:48.656059 21752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:47:48.656922 21901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:40849 every 8 connection(s)
I20260928 06:47:48.663539 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 21752
I20260928 06:47:48.664062 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:36585
--local_ip_for_outbound_sockets=127.7.48.2
--tserver_master_addrs=127.7.48.62:42645
--webserver_port=42761
--webserver_interface=127.7.48.2
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:47:48.665226 21794 log.cc:824] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4: Log is configured to *not* fsync() on all Append() calls
Sep 28 06:47:48 dist-test-slave-twl6 krb5kdc[21130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578068, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:47:48.675052 21794 tablet_bootstrap.cc:492] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:47:48.675540 21794 tablet_bootstrap.cc:492] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4: Bootstrap complete.
I20260928 06:47:48.677102 21794 ts_tablet_manager.cc:1397] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4: Time spent bootstrapping tablet: real 0.033s user 0.019s sys 0.012s
I20260928 06:47:48.677650 21905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:48.658289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.1:54541)
0928 06:47:48.659022 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:48.659077 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:48.659923 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:48.661073 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:48.661081 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:48.661509 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:47:48.661974 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:48.661988 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.663268 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.663274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:48.664686 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:48.664700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.665829 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.665833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:48.666037 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:48.666708 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:47:48.666728 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:47:48.672307 (+ 5579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:47:48.674459 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:48.674465 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:48.674478 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:48.674751 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:48.675099 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:48.675102 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:48.675104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:48.675296 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:48.675946 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:47:48.675952 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:47:48.676215 (+ 263us) client_negotiation.cc:769] Sending connection context
0928 06:47:48.676417 (+ 202us) client_negotiation.cc:240] Negotiation successful
0928 06:47:48.676742 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":136,"threads_started":1}
I20260928 06:47:48.677650 21904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:48.658237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:54541 (local address 127.7.48.62:42645)
0928 06:47:48.658741 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:48.658760 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:48.659978 (+ 1218us) server_negotiation.cc:407] Connection header received
0928 06:47:48.660366 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:48.660377 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:48.660670 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:48.661018 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:48.662155 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.663067 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.664846 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.665654 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.665852 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:48.672515 (+ 6663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:48.672520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:48.672532 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:48.672562 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:48.674318 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:48.674880 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:48.674885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:48.674888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:48.674999 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:48.675404 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:48.675407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:48.675409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:48.675808 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:48.675974 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:48.676538 (+ 564us) server_negotiation.cc:299] Negotiation successful
0928 06:47:48.676806 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":115,"threads_started":1}
I20260928 06:47:48.680223 21902 heartbeater.cc:347] Connected to a master server at 127.7.48.62:42645
I20260928 06:47:48.680459 21902 heartbeater.cc:464] Registering TS with master...
I20260928 06:47:48.680039 21794 raft_consensus.cc:348] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } } peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } }
I20260928 06:47:48.680569 21794 raft_consensus.cc:729] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bf016438a3e49c2828bdb251e4a1cc4, State: Initialized, Role: FOLLOWER
I20260928 06:47:48.680991 21902 heartbeater.cc:511] Master 127.7.48.62:42645 requested a full tablet report, sending...
I20260928 06:47:48.681064 21794 consensus_queue.cc:260] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } } peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } }
I20260928 06:47:48.682098 21794 ts_tablet_manager.cc:1428] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 06:47:48.683375 21713 ts_manager.cc:194] Registered new tserver with Master: 1bf016438a3e49c2828bdb251e4a1cc4 (127.7.48.1:40849)
I20260928 06:47:48.687060 21713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.1@KRBTEST.COM'} at 127.7.48.1:54541
I20260928 06:47:48.689658 21902 heartbeater.cc:503] Master 127.7.48.62:42645 was elected leader, sending a full tablet report...
Sep 28 06:47:48 dist-test-slave-twl6 krb5kdc[21130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578068, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for kudu/127.7.48.1@KRBTEST.COM
I20260928 06:47:48.695941 21910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:48.686183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55760 (local address 127.7.48.1:40849)
0928 06:47:48.686827 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:48.686840 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:48.686860 (+ 20us) server_negotiation.cc:407] Connection header received
0928 06:47:48.686920 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:48.686925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:48.687115 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:48.687260 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:48.687997 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.688697 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.689590 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.690074 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.690178 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:48.692634 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:48.692642 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:48.692652 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:48.692670 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:48.694110 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:48.694637 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:48.694641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:48.694642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:48.694710 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:48.694962 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:48.694964 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:48.694965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:48.695196 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:48.695292 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:48.695645 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 06:47:48.695768 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":84,"threads_started":1}
I20260928 06:47:48.695955 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:48.686094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:40849 (local address 127.0.0.1:55760)
0928 06:47:48.686534 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:48.686585 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:48.686714 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:48.687312 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:48.687319 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:48.687747 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:47:48.687868 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:48.687876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.688811 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.688816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:48.689476 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:48.689485 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.690207 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.690225 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:48.690340 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:48.690925 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:47:48.690939 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:47:48.692499 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:47:48.694231 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:48.694240 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:48.694245 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:48.694526 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:48.694791 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:48.694794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:48.694795 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:48.694884 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:48.695306 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:47:48.695311 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:47:48.695432 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 06:47:48.695581 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 06:47:48.695761 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":129,"threads_started":1}
I20260928 06:47:48.697157 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a0f55d777ae4c0b9f75586dbf1c140a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:47:48 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:55760
I20260928 06:47:48.697937 21911 tablet_replica.cc:330] stopping tablet replica
I20260928 06:47:48.698220 21911 raft_consensus.cc:2229] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:47:48.698467 21911 raft_consensus.cc:2258] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:47:48.699363 21911 ts_tablet_manager.cc:1905] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:47:48.702757 21911 ts_tablet_manager.cc:1918] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:47:48.702864 21911 log.cc:1196] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/wal/wals/3a0f55d777ae4c0b9f75586dbf1c140a
I20260928 06:47:48.703140 21911 ts_tablet_manager.cc:1939] T 3a0f55d777ae4c0b9f75586dbf1c140a P 1bf016438a3e49c2828bdb251e4a1cc4: Deleting consensus metadata
I20260928 06:47:48.704732 21701 catalog_manager.cc:4841] TS 1bf016438a3e49c2828bdb251e4a1cc4 (127.7.48.1:40849): tablet 3a0f55d777ae4c0b9f75586dbf1c140a (table test-table [id=be0d4e97918843699c24a2d04b18c814]) successfully deleted
W20260928 06:47:48.782081 21906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:47:48.782334 21906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:47:48.782395 21906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:47:48.786345 21906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:47:48.786401 21906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:47:48.786456 21906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:47:48.791342 21906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:36585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=42761
--tserver_master_addrs=127.7.48.62:42645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:47:48.792341 21906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:47:48.793291 21906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:47:48.804090 21916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:48.804087 21919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:48.804085 21917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:48.804241 21906 server_base.cc:1034] running on GCE node
I20260928 06:47:48.804755 21906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:47:48.805420 21906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:47:48.806636 21906 hybrid_clock.cc:648] HybridClock initialized: now 1790578068806625 us; error 30 us; skew 500 ppm
Sep 28 06:47:48 dist-test-slave-twl6 krb5kdc[21130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578068, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:48.809435 21906 init.cc:377] Logged in from keytab as kudu/127.7.48.2@KRBTEST.COM (short username kudu)
I20260928 06:47:48.810338 21906 webserver.cc:460] Webserver started at http://127.7.48.2:42761/ using document root <none> and password file <none>
I20260928 06:47:48.810904 21906 fs_manager.cc:359] Metadata directory not provided
I20260928 06:47:48.811007 21906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:47:48.814827 21906 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:47:48.816943 21926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:48.818156 21906 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:47:48.818302 21906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "09b49b0168804bdea2d1e25832aab4e8"
format_stamp: "Formatted at 2026-09-28 06:47:47 on dist-test-slave-twl6"
server_key: "4b48b55d1a375d523203a34b0db045a3"
server_key_iv: "4c2de843581fdf7f7cce91f7e4118026"
server_key_version: "encryptionkey@0"
I20260928 06:47:48.818845 21906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:47:48.835615 21906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:47:48.838990 21906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:47:48.839195 21906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:47:48.839794 21906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:47:48.840623 21906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:47:48.840689 21906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:48.840757 21906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:47:48.840806 21906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:48.849891 21906 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:36585
I20260928 06:47:48.849920 22039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:36585 every 8 connection(s)
I20260928 06:47:48.850939 21906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:47:48.852159 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 21906
I20260928 06:47:48.852589 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:35521
--local_ip_for_outbound_sockets=127.7.48.3
--tserver_master_addrs=127.7.48.62:42645
--webserver_port=45055
--webserver_interface=127.7.48.3
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:47:48 dist-test-slave-twl6 krb5kdc[21130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578068, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:47:48.864835 21904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:48.853008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:48263 (local address 127.7.48.62:42645)
0928 06:47:48.853186 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:48.853190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:48.854022 (+ 832us) server_negotiation.cc:407] Connection header received
0928 06:47:48.855379 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:48.855382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:48.855438 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:48.855553 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:48.856802 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.857328 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.858399 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.858643 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.858753 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:48.861281 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:48.861284 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:48.861286 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:48.861313 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:48.862792 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:48.863288 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:48.863292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:48.863294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:48.863373 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:48.863703 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:48.863707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:48.863709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:48.863935 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:48.864043 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:48.864533 (+ 490us) server_negotiation.cc:299] Negotiation successful
0928 06:47:48.864661 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 06:47:48.865677 22042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:48.853392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.2:48263)
0928 06:47:48.853860 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:48.853911 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:48.855146 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:48.855730 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:48.855737 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:48.856181 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:47:48.856654 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:48.856669 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.857446 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.857449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:48.858258 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:48.858266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:48.858792 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:48.858796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:48.858963 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:48.859566 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:47:48.859582 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:47:48.861106 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:47:48.862911 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:48.862917 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:48.862931 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:48.863186 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:48.863464 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:48.863467 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:48.863469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:48.863608 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:48.864044 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:47:48.864051 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:47:48.864298 (+ 247us) client_negotiation.cc:769] Sending connection context
0928 06:47:48.864485 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 06:47:48.864803 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":145,"threads_started":1}
I20260928 06:47:48.867027 22040 heartbeater.cc:347] Connected to a master server at 127.7.48.62:42645
I20260928 06:47:48.867278 22040 heartbeater.cc:464] Registering TS with master...
I20260928 06:47:48.867865 22040 heartbeater.cc:511] Master 127.7.48.62:42645 requested a full tablet report, sending...
I20260928 06:47:48.869051 21713 ts_manager.cc:194] Registered new tserver with Master: 09b49b0168804bdea2d1e25832aab4e8 (127.7.48.2:36585)
I20260928 06:47:48.869832 21713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.2@KRBTEST.COM'} at 127.7.48.2:48263
W20260928 06:47:48.962271 22043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:47:48.962528 22043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:47:48.962567 22043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:47:48.966395 22043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:47:48.966445 22043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:47:48.966501 22043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:47:48.971566 22043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:35521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=45055
--tserver_master_addrs=127.7.48.62:42645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:47:48.972611 22043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:47:48.973626 22043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:47:48.982869 22048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:48.982882 22051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:48.982986 22043 server_base.cc:1034] running on GCE node
W20260928 06:47:48.982847 22049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:48.983511 22043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:47:48.984165 22043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:47:48.985321 22043 hybrid_clock.cc:648] HybridClock initialized: now 1790578068985290 us; error 46 us; skew 500 ppm
Sep 28 06:47:48 dist-test-slave-twl6 krb5kdc[21130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578068, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:48.987943 22043 init.cc:377] Logged in from keytab as kudu/127.7.48.3@KRBTEST.COM (short username kudu)
I20260928 06:47:48.988812 22043 webserver.cc:460] Webserver started at http://127.7.48.3:45055/ using document root <none> and password file <none>
I20260928 06:47:48.989363 22043 fs_manager.cc:359] Metadata directory not provided
I20260928 06:47:48.989454 22043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:47:48.993091 22043 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:47:48.995097 22058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:48.996352 22043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:47:48.996483 22043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "f0393ad3546d48b6ac2a6a8243a00244"
format_stamp: "Formatted at 2026-09-28 06:47:47 on dist-test-slave-twl6"
server_key: "912d493adc0818cb3ddb8a118acbb522"
server_key_iv: "b041cd7707319e483b9c8edb8f366a5a"
server_key_version: "encryptionkey@0"
I20260928 06:47:48.997000 22043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:47:49.028194 22043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:47:49.030910 22043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:47:49.031102 22043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:47:49.031679 22043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:47:49.032500 22043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:47:49.032567 22043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:49.032634 22043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:47:49.032680 22043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:49.041354 22043 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:35521
I20260928 06:47:49.041380 22171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:35521 every 8 connection(s)
I20260928 06:47:49.042294 22043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:47:49.049265 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 22043
Sep 28 06:47:49 dist-test-slave-twl6 krb5kdc[21130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578068, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:47:49.057021 21904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:49.044085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:51473 (local address 127.7.48.62:42645)
0928 06:47:49.044231 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:49.044235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:49.045158 (+ 923us) server_negotiation.cc:407] Connection header received
0928 06:47:49.046432 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:49.046436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:49.046492 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:49.046566 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:49.047669 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.048227 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.049603 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.049933 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.050065 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:49.052910 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:49.052913 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:49.052916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:49.052941 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:49.054771 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:49.055301 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:49.055305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:49.055306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:49.055384 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:49.055767 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:49.055769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:49.055770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:49.056005 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:49.056117 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:49.056670 (+ 553us) server_negotiation.cc:299] Negotiation successful
0928 06:47:49.056847 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:47:49.057817 22174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:49.044572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.3:51473)
0928 06:47:49.045010 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:49.045051 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:49.046192 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:49.046734 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:49.046741 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:49.047080 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:47:49.047521 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:49.047551 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.048333 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.048335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:49.049491 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:49.049502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.050107 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.050113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:49.050312 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:49.050996 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:47:49.051013 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:47:49.052757 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:47:49.054884 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:49.054889 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:49.054901 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:49.055201 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:49.055480 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:49.055486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:49.055491 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:49.055678 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:49.056117 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:47:49.056126 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:47:49.056394 (+ 268us) client_negotiation.cc:769] Sending connection context
0928 06:47:49.056614 (+ 220us) client_negotiation.cc:240] Negotiation successful
0928 06:47:49.056953 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":119,"threads_started":1}
I20260928 06:47:49.058604 22176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:49.049974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49844 (local address 127.7.48.62:42645)
0928 06:47:49.050320 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:49.050325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:49.050453 (+ 128us) server_negotiation.cc:407] Connection header received
0928 06:47:49.050555 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:49.050561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:49.050614 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:49.050723 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:49.051499 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.052306 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.053335 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.053674 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.053755 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:49.055230 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:49.055237 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:49.055241 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:49.055271 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:49.056901 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:49.057360 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:49.057364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:49.057366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:49.057451 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:49.057750 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:49.057755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:49.057757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:49.057984 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:49.058200 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:49.058341 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 06:47:49.058448 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":104,"threads_started":1}
I20260928 06:47:49.059151 22172 heartbeater.cc:347] Connected to a master server at 127.7.48.62:42645
I20260928 06:47:49.059412 22172 heartbeater.cc:464] Registering TS with master...
I20260928 06:47:49.059966 22172 heartbeater.cc:511] Master 127.7.48.62:42645 requested a full tablet report, sending...
I20260928 06:47:49.060916 21713 ts_manager.cc:194] Registered new tserver with Master: f0393ad3546d48b6ac2a6a8243a00244 (127.7.48.3:35521)
I20260928 06:47:49.061437 21713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.3@KRBTEST.COM'} at 127.7.48.3:51473
I20260928 06:47:49.071931 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:47:49.075933 22176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:49.072548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49846 (local address 127.7.48.62:42645)
0928 06:47:49.072699 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:49.072703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:49.072844 (+ 141us) server_negotiation.cc:407] Connection header received
0928 06:47:49.072982 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:49.072984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:49.073089 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:49.073169 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:47:49.073350 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.073842 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.074596 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.074803 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.074877 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:49.075124 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:47:49.075530 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:47:49.075619 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:49.075669 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 06:47:49.075719 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:47:49.076587 21713 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:47:49.079607 21713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:47:49.087211 21836 tablet_service.cc:1467] Processing CreateTablet for tablet 905bdedc038147ad9e66d7c15b990578 (DEFAULT_TABLE table=test-table [id=ffdb477656ff4ea2b83b1b9986373e14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:47:49.087822 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 905bdedc038147ad9e66d7c15b990578. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:47:49.090581 21794 tablet_bootstrap.cc:492] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4: Bootstrap starting.
I20260928 06:47:49.090703 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:49.086111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:36585 (local address 127.0.0.1:39064)
0928 06:47:49.086241 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:49.086257 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:49.086333 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:49.087235 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:49.087237 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:49.087253 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:47:49.087354 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:49.087360 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.088567 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.088573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:49.089562 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:49.089570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.090305 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.090309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:49.090391 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:49.090426 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 06:47:49.090514 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:47:49.090588 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 06:47:49.091290 22181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:49.086338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39064 (local address 127.7.48.2:36585)
0928 06:47:49.086710 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:49.086722 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:49.086746 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:47:49.086827 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:49.086833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:49.087063 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:49.087247 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:47:49.087476 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.088440 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.089678 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.090187 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.090305 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:49.090442 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:49.091057 (+ 615us) server_negotiation.cc:299] Negotiation successful
0928 06:47:49.091155 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":129,"threads_started":1}
I20260928 06:47:49.091382 22182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:49.086587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:35521 (local address 127.0.0.1:52280)
0928 06:47:49.086894 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:49.086907 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:49.086991 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:49.087681 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:49.087683 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:49.087701 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:47:49.087780 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:49.087785 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.088967 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.088970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:49.089997 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:49.090007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.091035 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.091038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:49.091134 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:49.091165 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 06:47:49.091206 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 06:47:49.091254 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":87,"threads_started":1}
I20260928 06:47:49.091522 21794 tablet_bootstrap.cc:654] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4: Neither blocks nor log segments found. Creating new log.
I20260928 06:47:49.092864 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:49.086863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52280 (local address 127.7.48.3:35521)
0928 06:47:49.087244 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:49.087256 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:49.087277 (+ 21us) server_negotiation.cc:407] Connection header received
0928 06:47:49.087345 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:49.087349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:49.087533 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:49.087688 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:47:49.087888 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.088823 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.090142 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.090866 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.092344 (+ 1478us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:49.092473 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:49.092652 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 06:47:49.092739 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":85,"threads_started":1}
I20260928 06:47:49.093195 21794 tablet_bootstrap.cc:492] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4: No bootstrap required, opened a new log
I20260928 06:47:49.093295 21794 ts_tablet_manager.cc:1397] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:47:49.093761 21794 raft_consensus.cc:348] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } } peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } }
I20260928 06:47:49.093875 21794 raft_consensus.cc:374] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:47:49.093911 21794 raft_consensus.cc:729] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bf016438a3e49c2828bdb251e4a1cc4, State: Initialized, Role: FOLLOWER
I20260928 06:47:49.094038 21794 consensus_queue.cc:260] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } } peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } }
I20260928 06:47:49.094282 21794 ts_tablet_manager.cc:1428] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:47:49.094357 21974 tablet_service.cc:1467] Processing CreateTablet for tablet 905bdedc038147ad9e66d7c15b990578 (DEFAULT_TABLE table=test-table [id=ffdb477656ff4ea2b83b1b9986373e14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:47:49.095202 21974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 905bdedc038147ad9e66d7c15b990578. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:47:49.095132 22106 tablet_service.cc:1467] Processing CreateTablet for tablet 905bdedc038147ad9e66d7c15b990578 (DEFAULT_TABLE table=test-table [id=ffdb477656ff4ea2b83b1b9986373e14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:47:49.095925 22106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 905bdedc038147ad9e66d7c15b990578. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:47:49.101984 22185 tablet_bootstrap.cc:492] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8: Bootstrap starting.
I20260928 06:47:49.102159 22186 tablet_bootstrap.cc:492] T 905bdedc038147ad9e66d7c15b990578 P f0393ad3546d48b6ac2a6a8243a00244: Bootstrap starting.
I20260928 06:47:49.104024 22186 tablet_bootstrap.cc:654] T 905bdedc038147ad9e66d7c15b990578 P f0393ad3546d48b6ac2a6a8243a00244: Neither blocks nor log segments found. Creating new log.
I20260928 06:47:49.104198 22185 tablet_bootstrap.cc:654] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8: Neither blocks nor log segments found. Creating new log.
I20260928 06:47:49.104851 22185 log.cc:824] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:47:49.105010 22186 log.cc:824] T 905bdedc038147ad9e66d7c15b990578 P f0393ad3546d48b6ac2a6a8243a00244: Log is configured to *not* fsync() on all Append() calls
I20260928 06:47:49.107018 22186 tablet_bootstrap.cc:492] T 905bdedc038147ad9e66d7c15b990578 P f0393ad3546d48b6ac2a6a8243a00244: No bootstrap required, opened a new log
I20260928 06:47:49.107014 22185 tablet_bootstrap.cc:492] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8: No bootstrap required, opened a new log
I20260928 06:47:49.107228 22185 ts_tablet_manager.cc:1397] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 06:47:49.107239 22186 ts_tablet_manager.cc:1397] T 905bdedc038147ad9e66d7c15b990578 P f0393ad3546d48b6ac2a6a8243a00244: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 06:47:49.109885 22185 raft_consensus.cc:348] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } } peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } }
I20260928 06:47:49.109885 22186 raft_consensus.cc:348] T 905bdedc038147ad9e66d7c15b990578 P f0393ad3546d48b6ac2a6a8243a00244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } } peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } }
I20260928 06:47:49.110088 22186 raft_consensus.cc:374] T 905bdedc038147ad9e66d7c15b990578 P f0393ad3546d48b6ac2a6a8243a00244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:47:49.110116 22185 raft_consensus.cc:374] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:47:49.110138 22186 raft_consensus.cc:729] T 905bdedc038147ad9e66d7c15b990578 P f0393ad3546d48b6ac2a6a8243a00244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0393ad3546d48b6ac2a6a8243a00244, State: Initialized, Role: FOLLOWER
I20260928 06:47:49.110172 22185 raft_consensus.cc:729] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09b49b0168804bdea2d1e25832aab4e8, State: Initialized, Role: FOLLOWER
I20260928 06:47:49.110528 22186 consensus_queue.cc:260] T 905bdedc038147ad9e66d7c15b990578 P f0393ad3546d48b6ac2a6a8243a00244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } } peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } }
I20260928 06:47:49.110529 22185 consensus_queue.cc:260] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } } peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } }
I20260928 06:47:49.111222 22040 heartbeater.cc:503] Master 127.7.48.62:42645 was elected leader, sending a full tablet report...
I20260928 06:47:49.111287 22172 heartbeater.cc:503] Master 127.7.48.62:42645 was elected leader, sending a full tablet report...
I20260928 06:47:49.111428 22186 ts_tablet_manager.cc:1428] T 905bdedc038147ad9e66d7c15b990578 P f0393ad3546d48b6ac2a6a8243a00244: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260928 06:47:49.112547 22185 ts_tablet_manager.cc:1428] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260928 06:47:49.158118 21903 tablet.cc:2369] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:47:49.166126 22189 raft_consensus.cc:482] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:47:49.166368 22189 raft_consensus.cc:504] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } } peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } }
I20260928 06:47:49.167457 22189 leader_election.cc:288] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0393ad3546d48b6ac2a6a8243a00244 (127.7.48.3:35521), 1bf016438a3e49c2828bdb251e4a1cc4 (127.7.48.1:40849)
I20260928 06:47:49.172288 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:49.167787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:57207 (local address 127.7.48.3:35521)
0928 06:47:49.167954 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:49.167960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:49.167978 (+ 18us) server_negotiation.cc:407] Connection header received
0928 06:47:49.168063 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:49.168066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:49.168119 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:49.168204 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:47:49.168596 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.169315 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.170989 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.171613 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.171742 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:49.171804 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:49.172089 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 06:47:49.172166 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:47:49.172290 21910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:49.167951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:42491 (local address 127.7.48.1:40849)
0928 06:47:49.168113 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:49.168118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:49.168226 (+ 108us) server_negotiation.cc:407] Connection header received
0928 06:47:49.168361 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:49.168363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:49.168413 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:49.168486 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:47:49.168753 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.169442 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.170989 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.171657 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.171756 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:49.171959 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:49.172085 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 06:47:49.172162 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 06:47:49.172317 22042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:49.167698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:35521 (local address 127.7.48.2:57207)
0928 06:47:49.167824 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:49.167841 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:49.167961 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:49.168314 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:49.168316 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:49.168332 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:47:49.168449 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:49.168454 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.169421 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.169423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:49.170858 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:49.170869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.171774 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.171778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:49.171940 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:49.171971 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 06:47:49.172048 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 06:47:49.172165 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":140}
I20260928 06:47:49.172327 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:49.167698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:40849 (local address 127.7.48.2:42491)
0928 06:47:49.168146 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:49.168159 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:49.168255 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:49.168528 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:49.168533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:49.168555 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:47:49.168658 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:49.168662 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.169535 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.169537 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:49.170858 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:49.170869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.171774 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.171778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:49.171928 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:49.171975 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 06:47:49.172048 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:47:49.172165 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":197,"thread_start_us":150,"threads_started":1}
I20260928 06:47:49.173102 22126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "905bdedc038147ad9e66d7c15b990578" candidate_uuid: "09b49b0168804bdea2d1e25832aab4e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0393ad3546d48b6ac2a6a8243a00244" is_pre_election: true
I20260928 06:47:49.173091 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "905bdedc038147ad9e66d7c15b990578" candidate_uuid: "09b49b0168804bdea2d1e25832aab4e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" is_pre_election: true
I20260928 06:47:49.173420 22126 raft_consensus.cc:2454] T 905bdedc038147ad9e66d7c15b990578 P f0393ad3546d48b6ac2a6a8243a00244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09b49b0168804bdea2d1e25832aab4e8 in term 0.
I20260928 06:47:49.173420 21856 raft_consensus.cc:2454] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09b49b0168804bdea2d1e25832aab4e8 in term 0.
I20260928 06:47:49.173929 21928 leader_election.cc:302] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09b49b0168804bdea2d1e25832aab4e8, f0393ad3546d48b6ac2a6a8243a00244; no voters:
I20260928 06:47:49.174160 22189 raft_consensus.cc:2781] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:47:49.174261 22189 raft_consensus.cc:482] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:47:49.174324 22189 raft_consensus.cc:3037] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:47:49.175384 22189 raft_consensus.cc:504] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } } peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } }
I20260928 06:47:49.175745 22189 leader_election.cc:288] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8 [CANDIDATE]: Term 1 election: Requested vote from peers f0393ad3546d48b6ac2a6a8243a00244 (127.7.48.3:35521), 1bf016438a3e49c2828bdb251e4a1cc4 (127.7.48.1:40849)
I20260928 06:47:49.176139 22126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "905bdedc038147ad9e66d7c15b990578" candidate_uuid: "09b49b0168804bdea2d1e25832aab4e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0393ad3546d48b6ac2a6a8243a00244"
I20260928 06:47:49.176138 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "905bdedc038147ad9e66d7c15b990578" candidate_uuid: "09b49b0168804bdea2d1e25832aab4e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf016438a3e49c2828bdb251e4a1cc4"
I20260928 06:47:49.176271 22126 raft_consensus.cc:3037] T 905bdedc038147ad9e66d7c15b990578 P f0393ad3546d48b6ac2a6a8243a00244 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:47:49.176270 21856 raft_consensus.cc:3037] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:47:49.177533 22126 raft_consensus.cc:2454] T 905bdedc038147ad9e66d7c15b990578 P f0393ad3546d48b6ac2a6a8243a00244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09b49b0168804bdea2d1e25832aab4e8 in term 1.
I20260928 06:47:49.177538 21856 raft_consensus.cc:2454] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09b49b0168804bdea2d1e25832aab4e8 in term 1.
I20260928 06:47:49.177798 21928 leader_election.cc:302] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09b49b0168804bdea2d1e25832aab4e8, f0393ad3546d48b6ac2a6a8243a00244; no voters:
I20260928 06:47:49.177944 22189 raft_consensus.cc:2781] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:47:49.178180 22189 raft_consensus.cc:686] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8 [term 1 LEADER]: Becoming Leader. State: Replica: 09b49b0168804bdea2d1e25832aab4e8, State: Running, Role: LEADER
I20260928 06:47:49.178510 22189 consensus_queue.cc:237] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } } peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } }
I20260928 06:47:49.181592 21713 catalog_manager.cc:5495] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09b49b0168804bdea2d1e25832aab4e8 (127.7.48.2). New cstate: current_term: 1 leader_uuid: "09b49b0168804bdea2d1e25832aab4e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } health_report { overall_health: UNKNOWN } } }
I20260928 06:47:49.192032 22181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:49.188606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39074 (local address 127.7.48.2:36585)
0928 06:47:49.188726 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:49.188731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:49.188813 (+ 82us) server_negotiation.cc:407] Connection header received
0928 06:47:49.188987 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:49.188990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:49.189049 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:49.189155 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:47:49.189379 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.189867 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.190646 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:49.190862 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:49.190967 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:49.191260 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:47:49.191626 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:47:49.191760 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:49.191817 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 06:47:49.191869 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:47:49.197511 21856 raft_consensus.cc:1261] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 FOLLOWER]: Refusing update from remote peer 09b49b0168804bdea2d1e25832aab4e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:47:49.197513 22126 raft_consensus.cc:1261] T 905bdedc038147ad9e66d7c15b990578 P f0393ad3546d48b6ac2a6a8243a00244 [term 1 FOLLOWER]: Refusing update from remote peer 09b49b0168804bdea2d1e25832aab4e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:47:49.198158 22189 consensus_queue.cc:1035] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:47:49.198385 22192 consensus_queue.cc:1035] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:47:49.212828 21714 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:47:49.213013 21714 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:47:49.215420 21714 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72: Sending DeleteTablet for 3 replicas of tablet 905bdedc038147ad9e66d7c15b990578
I20260928 06:47:49.216132 21974 tablet_service.cc:1514] Processing DeleteTablet for tablet 905bdedc038147ad9e66d7c15b990578 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:47:49 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:39064
I20260928 06:47:49.216297 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 21752
I20260928 06:47:49.216295 22106 tablet_service.cc:1514] Processing DeleteTablet for tablet 905bdedc038147ad9e66d7c15b990578 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:47:49 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:52280
I20260928 06:47:49.217247 22204 tablet_replica.cc:330] T 905bdedc038147ad9e66d7c15b990578 P f0393ad3546d48b6ac2a6a8243a00244: stopping tablet replica
I20260928 06:47:49.217540 22204 raft_consensus.cc:2229] T 905bdedc038147ad9e66d7c15b990578 P f0393ad3546d48b6ac2a6a8243a00244 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:47:49.217830 22204 raft_consensus.cc:2258] T 905bdedc038147ad9e66d7c15b990578 P f0393ad3546d48b6ac2a6a8243a00244 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:47:49.218771 22205 tablet_replica.cc:330] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8: stopping tablet replica
I20260928 06:47:49.219067 22205 raft_consensus.cc:2229] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:47:49.219331 22204 ts_tablet_manager.cc:1905] T 905bdedc038147ad9e66d7c15b990578 P f0393ad3546d48b6ac2a6a8243a00244: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:47:49.219485 22205 raft_consensus.cc:2258] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:47:49.220893 22205 ts_tablet_manager.cc:1905] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:47:49.222187 22204 ts_tablet_manager.cc:1918] T 905bdedc038147ad9e66d7c15b990578 P f0393ad3546d48b6ac2a6a8243a00244: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:47:49.222290 22204 log.cc:1196] T 905bdedc038147ad9e66d7c15b990578 P f0393ad3546d48b6ac2a6a8243a00244: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/905bdedc038147ad9e66d7c15b990578
I20260928 06:47:49.222656 22204 ts_tablet_manager.cc:1939] T 905bdedc038147ad9e66d7c15b990578 P f0393ad3546d48b6ac2a6a8243a00244: Deleting consensus metadata
W20260928 06:47:49.223628 21701 connection.cc:535] client connection to 127.7.48.1:40849 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 06:47:49.223767 21701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:47:49.223804 22205 ts_tablet_manager.cc:1918] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:47:49.223917 21699 catalog_manager.cc:4841] TS f0393ad3546d48b6ac2a6a8243a00244 (127.7.48.3:35521): tablet 905bdedc038147ad9e66d7c15b990578 (table test-table [id=ffdb477656ff4ea2b83b1b9986373e14]) successfully deleted
I20260928 06:47:49.224053 22205 log.cc:1196] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/905bdedc038147ad9e66d7c15b990578
I20260928 06:47:49.224455 22205 ts_tablet_manager.cc:1939] T 905bdedc038147ad9e66d7c15b990578 P 09b49b0168804bdea2d1e25832aab4e8: Deleting consensus metadata
I20260928 06:47:49.224526 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 21906
I20260928 06:47:49.226639 22182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:49.226356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:40849 (local address 127.0.0.1:55774)
0928 06:47:49.226495 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:49.226569 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.1:40849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 06:47:49.226818 21701 catalog_manager.cc:4584] TS 1bf016438a3e49c2828bdb251e4a1cc4 (127.7.48.1:40849): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 905bdedc038147ad9e66d7c15b990578: Network error: Client connection negotiation failed: client connection to 127.7.48.1:40849: connect: Connection refused (error 111)
I20260928 06:47:49.230728 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 22043
I20260928 06:47:49.230886 22182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:49.230643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:36585 (local address 127.0.0.1:39090)
0928 06:47:49.230751 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:49.230791 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.2:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 06:47:49.235782 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 21681
I20260928 06:47:49.241473 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:42645
--webserver_interface=127.7.48.62
--webserver_port=44277
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:42645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:47:49.352267 22207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:47:49.352520 22207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:47:49.352558 22207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:47:49.356433 22207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:47:49.356479 22207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:47:49.356496 22207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:47:49.356511 22207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:47:49.356525 22207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:47:49.361465 22207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:42645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:42645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=44277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:47:49.362455 22207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:47:49.363353 22207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:47:49.371307 22215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:49.371300 22212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:49.371323 22213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:49.371574 22207 server_base.cc:1034] running on GCE node
I20260928 06:47:49.372260 22207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:47:49.373221 22207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:47:49.374397 22207 hybrid_clock.cc:648] HybridClock initialized: now 1790578069374366 us; error 64 us; skew 500 ppm
Sep 28 06:47:49 dist-test-slave-twl6 krb5kdc[21130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578069, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:49.377000 22207 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:47:49.377816 22207 webserver.cc:460] Webserver started at http://127.7.48.62:44277/ using document root <none> and password file <none>
I20260928 06:47:49.378341 22207 fs_manager.cc:359] Metadata directory not provided
I20260928 06:47:49.378427 22207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:47:49.382059 22207 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:47:49.384127 22222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:49.385277 22207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:47:49.385412 22207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "f139cf2a2416490abb75c93f5952fd72"
format_stamp: "Formatted at 2026-09-28 06:47:47 on dist-test-slave-twl6"
server_key: "a2603e8e989e8be342e7c758b288ade8"
server_key_iv: "a7ce030eba781f58374321d293ff897e"
server_key_version: "encryptionkey@0"
I20260928 06:47:49.385946 22207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:47:49.418161 22207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:47:49.421788 22207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:47:49.421988 22207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:47:49.428930 22207 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:42645
I20260928 06:47:49.429231 22274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:42645 every 8 connection(s)
I20260928 06:47:49.429993 22207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:47:49.434123 22275 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:47:49.435628 22275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72: Bootstrap starting.
I20260928 06:47:49.437772 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 22207
I20260928 06:47:49.438235 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:40849
--local_ip_for_outbound_sockets=127.7.48.1
--tserver_master_addrs=127.7.48.62:42645
--webserver_port=38095
--webserver_interface=127.7.48.1
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:47:49.453024 22275 log.cc:824] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72: Log is configured to *not* fsync() on all Append() calls
I20260928 06:47:49.463758 22275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:47:49.464133 22275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72: Bootstrap complete.
I20260928 06:47:49.468044 22275 raft_consensus.cc:348] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f139cf2a2416490abb75c93f5952fd72" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42645 } }
I20260928 06:47:49.468449 22275 raft_consensus.cc:729] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f139cf2a2416490abb75c93f5952fd72, State: Initialized, Role: FOLLOWER
I20260928 06:47:49.469025 22275 consensus_queue.cc:260] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f139cf2a2416490abb75c93f5952fd72" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42645 } }
I20260928 06:47:49.469139 22275 raft_consensus.cc:388] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:47:49.469238 22275 raft_consensus.cc:482] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:47:49.469359 22275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [term 2 FOLLOWER]: Advancing to term 3
I20260928 06:47:49.471103 22275 raft_consensus.cc:504] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f139cf2a2416490abb75c93f5952fd72" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42645 } }
I20260928 06:47:49.471422 22275 leader_election.cc:302] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f139cf2a2416490abb75c93f5952fd72; no voters:
I20260928 06:47:49.471758 22275 leader_election.cc:288] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 06:47:49.471837 22279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 06:47:49.472067 22279 raft_consensus.cc:686] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [term 3 LEADER]: Becoming Leader. State: Replica: f139cf2a2416490abb75c93f5952fd72, State: Running, Role: LEADER
I20260928 06:47:49.472342 22279 consensus_queue.cc:237] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f139cf2a2416490abb75c93f5952fd72" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42645 } }
I20260928 06:47:49.472929 22275 sys_catalog.cc:564] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:47:49.473250 22279 sys_catalog.cc:455] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f139cf2a2416490abb75c93f5952fd72. Latest consensus state: current_term: 3 leader_uuid: "f139cf2a2416490abb75c93f5952fd72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f139cf2a2416490abb75c93f5952fd72" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42645 } } }
I20260928 06:47:49.473363 22279 sys_catalog.cc:458] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [sys.catalog]: This master's current role is: LEADER
I20260928 06:47:49.473765 22283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:47:49.473770 22280 sys_catalog.cc:455] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f139cf2a2416490abb75c93f5952fd72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f139cf2a2416490abb75c93f5952fd72" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42645 } } }
I20260928 06:47:49.474032 22280 sys_catalog.cc:458] T 00000000000000000000000000000000 P f139cf2a2416490abb75c93f5952fd72 [sys.catalog]: This master's current role is: LEADER
I20260928 06:47:49.478636 22283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:47:49.480175 22283 catalog_manager.cc:1259] Loaded cluster ID: d638604e7cee4b45b524b2ff27dd0396
I20260928 06:47:49.480228 22283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 06:47:49.481408 22283 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa244db99ad google::LogMessage::Fail() at ??:0
@ 0x7fa244dbc88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa244db94de google::LogMessage::Flush() at ??:0
@ 0x7fa244dbaa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa24f6f8fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fa24f6f9d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa24f6f6579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fa24f73a640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa24700b3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa24700bc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fa24700d3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa246ffc512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa2486406db start_thread at ??:0
@ 0x7fa2438b761f clone at ??:0
W20260928 06:47:49.558915 22277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:47:49.559188 22277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:47:49.559288 22277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:47:49.563035 22277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:47:49.563117 22277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:47:49.563222 22277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:47:49.568360 22277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:40849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=38095
--tserver_master_addrs=127.7.48.62:42645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:47:49.569504 22277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:47:49.570539 22277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:47:49.579835 22316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:49.579917 22277 server_base.cc:1034] running on GCE node
W20260928 06:47:49.580076 22317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:49.579831 22319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:49.580502 22277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:47:49.581164 22277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:47:49.582338 22277 hybrid_clock.cc:648] HybridClock initialized: now 1790578069582308 us; error 50 us; skew 500 ppm
Sep 28 06:47:49 dist-test-slave-twl6 krb5kdc[21130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578069, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:49.585086 22277 init.cc:377] Logged in from keytab as kudu/127.7.48.1@KRBTEST.COM (short username kudu)
I20260928 06:47:49.585989 22277 webserver.cc:460] Webserver started at http://127.7.48.1:38095/ using document root <none> and password file <none>
I20260928 06:47:49.586583 22277 fs_manager.cc:359] Metadata directory not provided
I20260928 06:47:49.586670 22277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:47:49.590698 22277 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:47:49.592738 22326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:49.593958 22277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 06:47:49.594087 22277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "1bf016438a3e49c2828bdb251e4a1cc4"
format_stamp: "Formatted at 2026-09-28 06:47:47 on dist-test-slave-twl6"
server_key: "ac34a05a8e13f170e2e8885403fc9f13"
server_key_iv: "8cfde00221215735cf050cda0ce9c0cc"
server_key_version: "encryptionkey@0"
I20260928 06:47:49.594617 22277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:47:49.613946 22277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:47:49.617445 22277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:47:49.617640 22277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:47:49.618297 22277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:47:49.619467 22333 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:47:49.624775 22277 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:47:49.624850 22277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260928 06:47:49.624945 22277 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:47:49.626637 22277 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:47:49.626704 22277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 06:47:49.626993 22333 tablet_bootstrap.cc:492] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4: Bootstrap starting.
I20260928 06:47:49.637606 22277 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:40849
I20260928 06:47:49.637657 22440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:40849 every 8 connection(s)
I20260928 06:47:49.638597 22277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:47:49.644997 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 22277
I20260928 06:47:49.645485 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:36585
--local_ip_for_outbound_sockets=127.7.48.2
--tserver_master_addrs=127.7.48.62:42645
--webserver_port=42761
--webserver_interface=127.7.48.2
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:47:49.648741 22333 log.cc:824] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4: Log is configured to *not* fsync() on all Append() calls
I20260928 06:47:49.658740 22333 tablet_bootstrap.cc:492] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:47:49.659209 22333 tablet_bootstrap.cc:492] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4: Bootstrap complete.
I20260928 06:47:49.660724 22333 ts_tablet_manager.cc:1397] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4: Time spent bootstrapping tablet: real 0.034s user 0.022s sys 0.010s
I20260928 06:47:49.664201 22333 raft_consensus.cc:348] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } } peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } }
I20260928 06:47:49.664786 22333 raft_consensus.cc:729] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bf016438a3e49c2828bdb251e4a1cc4, State: Initialized, Role: FOLLOWER
I20260928 06:47:49.665366 22333 consensus_queue.cc:260] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } } peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } }
I20260928 06:47:49.666553 22333 ts_tablet_manager.cc:1428] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20260928 06:47:49.696405 22443 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:47:49.640569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.1:36493)
0928 06:47:49.641090 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:49.641145 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:49.641956 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:49.695368 (+ 53412us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":168,"threads_started":1}
W20260928 06:47:49.697036 22441 heartbeater.cc:644] Failed to heartbeat to 127.7.48.62:42645 (0 consecutive failures): Network error: Failed to ping master at 127.7.48.62:42645: Client connection negotiation failed: client connection to 127.7.48.62:42645: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:47:49.697808 22443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:49.697510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.1:47773)
0928 06:47:49.697648 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:49.697724 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 06:47:49.698484 22443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:49.698319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.1:36063)
0928 06:47:49.698400 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:49.698439 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 06:47:49.699103 22441 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:47:49.762763 22444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:47:49.763038 22444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:47:49.763128 22444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:47:49.767113 22444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:47:49.767177 22444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:47:49.767267 22444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:47:49.772403 22444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:36585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=42761
--tserver_master_addrs=127.7.48.62:42645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:47:49.773406 22444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:47:49.774528 22444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:47:49.784067 22451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:49.784094 22454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:49.784090 22452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:49.784332 22444 server_base.cc:1034] running on GCE node
I20260928 06:47:49.784855 22444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:47:49.785454 22444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:47:49.786609 22444 hybrid_clock.cc:648] HybridClock initialized: now 1790578069786594 us; error 29 us; skew 500 ppm
Sep 28 06:47:49 dist-test-slave-twl6 krb5kdc[21130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578069, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:49.789188 22444 init.cc:377] Logged in from keytab as kudu/127.7.48.2@KRBTEST.COM (short username kudu)
I20260928 06:47:49.790067 22444 webserver.cc:460] Webserver started at http://127.7.48.2:42761/ using document root <none> and password file <none>
I20260928 06:47:49.790614 22444 fs_manager.cc:359] Metadata directory not provided
I20260928 06:47:49.790709 22444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:47:49.794404 22444 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:47:49.796522 22461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:49.797763 22444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:47:49.797889 22444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "09b49b0168804bdea2d1e25832aab4e8"
format_stamp: "Formatted at 2026-09-28 06:47:47 on dist-test-slave-twl6"
server_key: "4b48b55d1a375d523203a34b0db045a3"
server_key_iv: "4c2de843581fdf7f7cce91f7e4118026"
server_key_version: "encryptionkey@0"
I20260928 06:47:49.798561 22444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:47:49.812853 22444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:47:49.815652 22444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:47:49.815855 22444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:47:49.816509 22444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:47:49.817338 22444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:47:49.817406 22444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:49.817474 22444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:47:49.817516 22444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:49.826439 22444 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:36585
I20260928 06:47:49.826474 22574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:36585 every 8 connection(s)
I20260928 06:47:49.827462 22444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:47:49.832026 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:49.829967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.2:43647)
0928 06:47:49.830543 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:49.830721 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":145,"threads_started":1}
W20260928 06:47:49.832444 22575 heartbeater.cc:644] Failed to heartbeat to 127.7.48.62:42645 (0 consecutive failures): Network error: Failed to ping master at 127.7.48.62:42645: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
I20260928 06:47:49.832834 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 22444
I20260928 06:47:49.833087 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:49.832868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.2:41331)
0928 06:47:49.832993 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:49.833037 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 06:47:49.833279 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:35521
--local_ip_for_outbound_sockets=127.7.48.3
--tserver_master_addrs=127.7.48.62:42645
--webserver_port=45055
--webserver_interface=127.7.48.3
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:47:49.833731 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:49.833555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.2:36485)
0928 06:47:49.833645 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:49.833685 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 06:47:49.833928 22575 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:47:49.943944 22578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:47:49.944208 22578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:47:49.944247 22578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:47:49.948221 22578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:47:49.948283 22578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:47:49.948381 22578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:47:49.953648 22578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:35521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=45055
--tserver_master_addrs=127.7.48.62:42645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:47:49.954741 22578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:47:49.955842 22578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:47:49.965754 22584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:47:49.965762 22586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:49.965837 22578 server_base.cc:1034] running on GCE node
W20260928 06:47:49.965770 22583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:47:49.966367 22578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:47:49.966988 22578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:47:49.968145 22578 hybrid_clock.cc:648] HybridClock initialized: now 1790578069968113 us; error 43 us; skew 500 ppm
Sep 28 06:47:49 dist-test-slave-twl6 krb5kdc[21130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578069, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:47:49.970844 22578 init.cc:377] Logged in from keytab as kudu/127.7.48.3@KRBTEST.COM (short username kudu)
I20260928 06:47:49.971632 22578 webserver.cc:460] Webserver started at http://127.7.48.3:45055/ using document root <none> and password file <none>
I20260928 06:47:49.972230 22578 fs_manager.cc:359] Metadata directory not provided
I20260928 06:47:49.972294 22578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:47:49.976009 22578 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260928 06:47:49.978015 22593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:49.979210 22578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:47:49.979301 22578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "f0393ad3546d48b6ac2a6a8243a00244"
format_stamp: "Formatted at 2026-09-28 06:47:47 on dist-test-slave-twl6"
server_key: "912d493adc0818cb3ddb8a118acbb522"
server_key_iv: "b041cd7707319e483b9c8edb8f366a5a"
server_key_version: "encryptionkey@0"
I20260928 06:47:49.979825 22578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:47:50.024274 22578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:47:50.027132 22578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:47:50.027293 22578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:47:50.027858 22578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:47:50.028704 22578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:47:50.028775 22578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:50.028841 22578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:47:50.028895 22578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:47:50.037654 22578 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:35521
I20260928 06:47:50.037701 22706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:35521 every 8 connection(s)
I20260928 06:47:50.038614 22578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:47:50.040418 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 22578
I20260928 06:47:50.042719 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:50.040807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.3:59619)
0928 06:47:50.041304 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:50.041499 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":162,"threads_started":1}
W20260928 06:47:50.043166 22707 heartbeater.cc:644] Failed to heartbeat to 127.7.48.62:42645 (0 consecutive failures): Network error: Failed to ping master at 127.7.48.62:42645: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
I20260928 06:47:50.043758 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:50.043532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.3:48435)
0928 06:47:50.043645 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:50.043694 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 06:47:50.044340 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:50.044151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.3:55265)
0928 06:47:50.044248 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:50.044288 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 06:47:50.044504 22707 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:47:50.700088 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:50.699658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.1:60173)
0928 06:47:50.699917 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:50.699975 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":76,"threads_started":1}
I20260928 06:47:50.834864 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:50.834447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.2:41207)
0928 06:47:50.834706 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:50.834763 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":72,"threads_started":1}
I20260928 06:47:51.045481 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:51.045065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.3:33081)
0928 06:47:51.045303 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:51.045360 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":70,"threads_started":1}
I20260928 06:47:51.273494 22714 raft_consensus.cc:482] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:47:51.273761 22714 raft_consensus.cc:504] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } } peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } }
I20260928 06:47:51.274958 22714 leader_election.cc:288] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09b49b0168804bdea2d1e25832aab4e8 (127.7.48.2:36585), f0393ad3546d48b6ac2a6a8243a00244 (127.7.48.3:35521)
Sep 28 06:47:51 dist-test-slave-twl6 krb5kdc[21130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578069, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for kudu/127.7.48.2@KRBTEST.COM
Sep 28 06:47:51 dist-test-slave-twl6 krb5kdc[21130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578069, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for kudu/127.7.48.3@KRBTEST.COM
I20260928 06:47:51.288741 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:51.275180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:36585 (local address 127.7.48.1:43781)
0928 06:47:51.275484 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:51.275511 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:51.275628 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:51.277566 (+ 1938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:51.277569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:51.278142 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:47:51.278621 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:51.278636 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:51.279915 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:51.279919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:51.280812 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:51.280822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:51.281490 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:51.281493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:51.281695 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:51.282407 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:47:51.282419 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:47:51.284079 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:47:51.286157 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:51.286163 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:51.286177 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:51.286423 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:51.286803 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:51.286807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:51.286809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:51.286945 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:51.287688 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:47:51.287694 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:47:51.288018 (+ 324us) client_negotiation.cc:769] Sending connection context
0928 06:47:51.288229 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 06:47:51.288554 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":99,"thread_start_us":78,"threads_started":1}
I20260928 06:47:51.288849 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:51.275340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:43781 (local address 127.7.48.2:36585)
0928 06:47:51.275716 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:51.275744 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:51.275793 (+ 49us) server_negotiation.cc:407] Connection header received
0928 06:47:51.276756 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:51.276767 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:51.277195 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:51.277498 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:51.278785 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:51.279724 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:51.280934 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:51.281367 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:51.281533 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:51.284333 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:51.284338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:51.284350 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:51.284376 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:51.286024 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:51.286562 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:51.286568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:51.286572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:51.286704 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:51.287041 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:51.287045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:51.287047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:51.287568 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:51.287717 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:51.288347 (+ 630us) server_negotiation.cc:299] Negotiation successful
0928 06:47:51.288666 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":123,"threads_started":1}
I20260928 06:47:51.289288 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:51.275522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:42627 (local address 127.7.48.3:35521)
0928 06:47:51.275837 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0928 06:47:51.275856 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 06:47:51.275982 (+ 126us) server_negotiation.cc:407] Connection header received
0928 06:47:51.276881 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:47:51.276890 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:47:51.277215 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:47:51.277497 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:47:51.278784 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:51.279732 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:51.280912 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:47:51.281356 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:47:51.281526 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:51.284474 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:47:51.284478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:47:51.284489 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:47:51.284522 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:47:51.286364 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:51.286859 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:51.286864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:51.286868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:51.286972 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:47:51.287856 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:47:51.287860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:47:51.287861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:47:51.288359 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:47:51.288513 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 06:47:51.288853 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 06:47:51.289125 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":105,"threads_started":1}
I20260928 06:47:51.288965 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:51.275294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:35521 (local address 127.7.48.1:42627)
0928 06:47:51.275862 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:51.275897 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:47:51.275997 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:47:51.277554 (+ 1557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:47:51.277560 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:47:51.278146 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:47:51.278620 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:51.278635 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:51.279870 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:51.279901 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:51.280789 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:47:51.280799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:47:51.281480 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:47:51.281484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:47:51.281724 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:47:51.282363 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:47:51.282379 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:47:51.284292 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:47:51.286502 (+ 2210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:51.286509 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:51.286511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:51.286735 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:51.287391 (+ 656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:47:51.287394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:47:51.287396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:47:51.287486 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:47:51.288475 (+ 989us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:47:51.288478 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:47:51.288581 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 06:47:51.288685 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 06:47:51.288805 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":159,"spinlock_wait_cycles":6912,"thread_start_us":95,"threads_started":1}
W20260928 06:47:51.291532 22329 leader_election.cc:341] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09b49b0168804bdea2d1e25832aab4e8 (127.7.48.2:36585): Not found: Tablet not found: 905bdedc038147ad9e66d7c15b990578
W20260928 06:47:51.291706 22328 leader_election.cc:341] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0393ad3546d48b6ac2a6a8243a00244 (127.7.48.3:35521): Not found: Tablet not found: 905bdedc038147ad9e66d7c15b990578
I20260928 06:47:51.291815 22328 leader_election.cc:302] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bf016438a3e49c2828bdb251e4a1cc4; no voters: 09b49b0168804bdea2d1e25832aab4e8, f0393ad3546d48b6ac2a6a8243a00244
I20260928 06:47:51.292091 22714 raft_consensus.cc:2726] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:47:51.701169 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:51.700797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.1:41765)
0928 06:47:51.701005 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:51.701067 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260928 06:47:51.836089 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:51.835624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.2:37603)
0928 06:47:51.835935 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:51.835993 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":80,"threads_started":1}
I20260928 06:47:52.046901 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:52.046330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.3:44569)
0928 06:47:52.046687 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:52.046757 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":121,"threads_started":1}
I20260928 06:47:52.702350 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:52.701943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.1:37401)
0928 06:47:52.702186 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:52.702253 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":91,"threads_started":1}
I20260928 06:47:52.837342 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:52.836908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.2:43077)
0928 06:47:52.837177 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:52.837234 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":104,"threads_started":1}
I20260928 06:47:52.974115 22723 raft_consensus.cc:482] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:47:52.974277 22723 raft_consensus.cc:504] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } } peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } }
I20260928 06:47:52.974713 22723 leader_election.cc:288] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09b49b0168804bdea2d1e25832aab4e8 (127.7.48.2:36585), f0393ad3546d48b6ac2a6a8243a00244 (127.7.48.3:35521)
W20260928 06:47:52.975574 22329 leader_election.cc:341] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09b49b0168804bdea2d1e25832aab4e8 (127.7.48.2:36585): Not found: Tablet not found: 905bdedc038147ad9e66d7c15b990578
W20260928 06:47:52.975673 22328 leader_election.cc:341] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0393ad3546d48b6ac2a6a8243a00244 (127.7.48.3:35521): Not found: Tablet not found: 905bdedc038147ad9e66d7c15b990578
I20260928 06:47:52.975778 22328 leader_election.cc:302] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bf016438a3e49c2828bdb251e4a1cc4; no voters: 09b49b0168804bdea2d1e25832aab4e8, f0393ad3546d48b6ac2a6a8243a00244
I20260928 06:47:52.975968 22723 raft_consensus.cc:2726] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:47:53.048070 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:53.047592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.3:58471)
0928 06:47:53.047865 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:53.047969 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":132,"threads_started":1}
I20260928 06:47:53.703553 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:53.703134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.1:58293)
0928 06:47:53.703384 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:53.703458 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":72,"threads_started":1}
I20260928 06:47:53.838544 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:53.838150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.2:52685)
0928 06:47:53.838403 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:53.838456 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":89,"threads_started":1}
I20260928 06:47:54.049149 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:54.048763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.3:45289)
0928 06:47:54.049002 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:54.049061 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":70,"threads_started":1}
I20260928 06:47:54.704866 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:54.704423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.1:43175)
0928 06:47:54.704708 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:54.704762 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":80,"threads_started":1}
I20260928 06:47:54.839812 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:54.839350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.2:47187)
0928 06:47:54.839669 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:54.839726 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":69,"threads_started":1}
I20260928 06:47:55.050472 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:55.049969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.3:35423)
0928 06:47:55.050291 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:55.050380 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":116,"threads_started":1}
I20260928 06:47:55.392314 22731 raft_consensus.cc:482] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:47:55.392458 22731 raft_consensus.cc:504] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } } peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } }
I20260928 06:47:55.392933 22731 leader_election.cc:288] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09b49b0168804bdea2d1e25832aab4e8 (127.7.48.2:36585), f0393ad3546d48b6ac2a6a8243a00244 (127.7.48.3:35521)
W20260928 06:47:55.393756 22328 leader_election.cc:341] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0393ad3546d48b6ac2a6a8243a00244 (127.7.48.3:35521): Not found: Tablet not found: 905bdedc038147ad9e66d7c15b990578
W20260928 06:47:55.393898 22329 leader_election.cc:341] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09b49b0168804bdea2d1e25832aab4e8 (127.7.48.2:36585): Not found: Tablet not found: 905bdedc038147ad9e66d7c15b990578
I20260928 06:47:55.393990 22329 leader_election.cc:302] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bf016438a3e49c2828bdb251e4a1cc4; no voters: 09b49b0168804bdea2d1e25832aab4e8, f0393ad3546d48b6ac2a6a8243a00244
I20260928 06:47:55.394150 22731 raft_consensus.cc:2726] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:47:55.705953 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:55.705545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.1:44407)
0928 06:47:55.705806 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:55.705859 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":70,"threads_started":1}
I20260928 06:47:55.840934 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:55.840521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.2:33373)
0928 06:47:55.840763 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:55.840833 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":85,"threads_started":1}
I20260928 06:47:56.051563 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:56.051169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.3:50285)
0928 06:47:56.051417 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:56.051470 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":91,"threads_started":1}
I20260928 06:47:56.707129 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:56.706692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.1:60645)
0928 06:47:56.706971 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:56.707025 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":93,"threads_started":1}
I20260928 06:47:56.841995 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:56.841603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.2:40167)
0928 06:47:56.841864 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:56.841913 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":65,"threads_started":1}
I20260928 06:47:57.052774 22737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:57.052319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.3:46429)
0928 06:47:57.052622 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:57.052679 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":92,"threads_started":1}
I20260928 06:47:57.708309 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:57.707870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.1:57129)
0928 06:47:57.708158 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:57.708212 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
I20260928 06:47:57.843061 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:57.842724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.2:50353)
0928 06:47:57.842927 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:57.842974 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":66,"threads_started":1}
I20260928 06:47:58.053879 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:58.053486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.3:56659)
0928 06:47:58.053741 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:58.053790 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":90,"threads_started":1}
I20260928 06:47:58.709535 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:58.709161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.1:48463)
0928 06:47:58.709399 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:58.709448 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":75,"threads_started":1}
I20260928 06:47:58.844080 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:58.843682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.2:57583)
0928 06:47:58.843943 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:58.843997 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":68,"threads_started":1}
I20260928 06:47:58.913956 22743 raft_consensus.cc:482] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:47:58.914080 22743 raft_consensus.cc:504] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } } peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } }
I20260928 06:47:58.914546 22743 leader_election.cc:288] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09b49b0168804bdea2d1e25832aab4e8 (127.7.48.2:36585), f0393ad3546d48b6ac2a6a8243a00244 (127.7.48.3:35521)
W20260928 06:47:58.915344 22329 leader_election.cc:341] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09b49b0168804bdea2d1e25832aab4e8 (127.7.48.2:36585): Not found: Tablet not found: 905bdedc038147ad9e66d7c15b990578
W20260928 06:47:58.915452 22328 leader_election.cc:341] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0393ad3546d48b6ac2a6a8243a00244 (127.7.48.3:35521): Not found: Tablet not found: 905bdedc038147ad9e66d7c15b990578
I20260928 06:47:58.915565 22328 leader_election.cc:302] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bf016438a3e49c2828bdb251e4a1cc4; no voters: 09b49b0168804bdea2d1e25832aab4e8, f0393ad3546d48b6ac2a6a8243a00244
I20260928 06:47:58.915745 22743 raft_consensus.cc:2726] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:47:59.055011 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:59.054621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.3:46477)
0928 06:47:59.054874 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:59.054923 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":102,"threads_started":1}
I20260928 06:47:59.710657 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:59.710290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.1:33665)
0928 06:47:59.710511 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:59.710557 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":78,"threads_started":1}
I20260928 06:47:59.845149 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:47:59.844773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.2:42877)
0928 06:47:59.845005 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 06:47:59.845057 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":72,"threads_started":1}
I20260928 06:48:00.056200 22747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:00.055742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.3:33439)
0928 06:48:00.056037 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:00.056097 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":92,"threads_started":1}
I20260928 06:48:00.712085 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:00.711552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.1:38711)
0928 06:48:00.711805 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:00.711862 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":74,"threads_started":1}
I20260928 06:48:00.846299 22749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:00.845879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.2:59511)
0928 06:48:00.846130 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:00.846189 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":77,"threads_started":1}
I20260928 06:48:01.057391 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:01.056967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.3:50337)
0928 06:48:01.057233 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:01.057293 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":74,"threads_started":1}
I20260928 06:48:01.713219 22751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:01.712824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.1:57871)
0928 06:48:01.713064 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:01.713122 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":76,"threads_started":1}
I20260928 06:48:01.847465 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:01.847074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.2:59701)
0928 06:48:01.847316 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:01.847371 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
I20260928 06:48:02.058889 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:02.058402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.3:49713)
0928 06:48:02.058701 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:02.058759 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":92,"threads_started":1}
I20260928 06:48:02.714424 22754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:02.714024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.1:39941)
0928 06:48:02.714268 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:02.714327 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":73,"threads_started":1}
I20260928 06:48:02.848647 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:02.848182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.2:47583)
0928 06:48:02.848443 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:02.848506 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":67,"threads_started":1}
I20260928 06:48:03.060267 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:03.059729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.3:48055)
0928 06:48:03.060117 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:03.060173 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":70,"threads_started":1}
I20260928 06:48:03.715547 22757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:03.715168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.1:52561)
0928 06:48:03.715402 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:03.715455 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":74,"threads_started":1}
I20260928 06:48:03.849733 22758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:03.849333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.2:49925)
0928 06:48:03.849594 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:03.849645 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":84,"threads_started":1}
I20260928 06:48:04.061308 22759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:04.060940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.3:35359)
0928 06:48:04.061165 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:04.061216 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":73,"threads_started":1}
I20260928 06:48:04.630230 22760 raft_consensus.cc:482] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:48:04.630434 22760 raft_consensus.cc:504] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b49b0168804bdea2d1e25832aab4e8" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36585 } } peers { permanent_uuid: "f0393ad3546d48b6ac2a6a8243a00244" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 35521 } } peers { permanent_uuid: "1bf016438a3e49c2828bdb251e4a1cc4" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40849 } }
I20260928 06:48:04.630877 22760 leader_election.cc:288] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09b49b0168804bdea2d1e25832aab4e8 (127.7.48.2:36585), f0393ad3546d48b6ac2a6a8243a00244 (127.7.48.3:35521)
W20260928 06:48:04.631659 22329 leader_election.cc:341] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09b49b0168804bdea2d1e25832aab4e8 (127.7.48.2:36585): Not found: Tablet not found: 905bdedc038147ad9e66d7c15b990578
W20260928 06:48:04.631765 22328 leader_election.cc:341] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0393ad3546d48b6ac2a6a8243a00244 (127.7.48.3:35521): Not found: Tablet not found: 905bdedc038147ad9e66d7c15b990578
I20260928 06:48:04.631951 22328 leader_election.cc:302] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bf016438a3e49c2828bdb251e4a1cc4; no voters: 09b49b0168804bdea2d1e25832aab4e8, f0393ad3546d48b6ac2a6a8243a00244
I20260928 06:48:04.632143 22760 raft_consensus.cc:2726] T 905bdedc038147ad9e66d7c15b990578 P 1bf016438a3e49c2828bdb251e4a1cc4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:48:04.716970 22761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:04.716585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.1:44311)
0928 06:48:04.716824 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:04.716876 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":76,"threads_started":1}
I20260928 06:48:04.851310 22762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:04.850831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:42645 (local address 127.7.48.2:47033)
0928 06:48:04.851087 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:04.851164 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":78,"threads_started":1}
I20260928 06:48:05.047230 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 22277
I20260928 06:48:05.053603 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 22444
I20260928 06:48:05.058843 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 22578
2026-09-28T06:48:05Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19132 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:48:05 dist-test-slave-twl6 krb5kdc[22765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:48:05 dist-test-slave-twl6 krb5kdc[22765](info): set up 2 sockets
Sep 28 06:48:05 dist-test-slave-twl6 krb5kdc[22765](info): commencing operation
krb5kdc: starting...
W20260928 06:48:06.044371 7360 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:48:06 dist-test-slave-twl6 krb5kdc[22765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:48:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:48:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:48:06.156551 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:33141
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:33141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:48:06.259730 22778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:48:06.260059 22778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:48:06.260140 22778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:48:06.264161 22778 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 06:48:06.264250 22778 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 06:48:06.264276 22778 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 06:48:06.264298 22778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:48:06.264343 22778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:48:06.264382 22778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:48:06.264420 22778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:48:06.264453 22778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:48:06.269662 22778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:33141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:33141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:48:06.270779 22778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:48:06.271792 22778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:48:06.280740 22786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:48:06.280752 22784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:48:06.280824 22778 server_base.cc:1034] running on GCE node
W20260928 06:48:06.280754 22783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:48:06.281651 22778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:48:06.282758 22778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:48:06.284003 22778 hybrid_clock.cc:648] HybridClock initialized: now 1790578086283971 us; error 50 us; skew 500 ppm
Sep 28 06:48:06 dist-test-slave-twl6 krb5kdc[22765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578086, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:48:06.291340 22778 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:48:06.292382 22778 webserver.cc:460] Webserver started at http://127.7.48.62:41089/ using document root <none> and password file <none>
I20260928 06:48:06.292966 22778 fs_manager.cc:359] Metadata directory not provided
I20260928 06:48:06.293058 22778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:48:06.293292 22778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:48:06.295343 22778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "52938bc4827343ae836c286f671af74f"
format_stamp: "Formatted at 2026-09-28 06:48:06 on dist-test-slave-twl6"
server_key: "4608f488d78d9b4f78aa11fbc9eec076"
server_key_iv: "2a3e05411162a03641ae8209db572de0"
server_key_version: "encryptionkey@0"
I20260928 06:48:06.295941 22778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "52938bc4827343ae836c286f671af74f"
format_stamp: "Formatted at 2026-09-28 06:48:06 on dist-test-slave-twl6"
server_key: "4608f488d78d9b4f78aa11fbc9eec076"
server_key_iv: "2a3e05411162a03641ae8209db572de0"
server_key_version: "encryptionkey@0"
I20260928 06:48:06.299749 22778 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 06:48:06.302251 22793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:06.303570 22778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:48:06.303710 22778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "52938bc4827343ae836c286f671af74f"
format_stamp: "Formatted at 2026-09-28 06:48:06 on dist-test-slave-twl6"
server_key: "4608f488d78d9b4f78aa11fbc9eec076"
server_key_iv: "2a3e05411162a03641ae8209db572de0"
server_key_version: "encryptionkey@0"
I20260928 06:48:06.303803 22778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:48:06.321194 22778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:48:06.324414 22778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:48:06.324620 22778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:48:06.331315 22778 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:33141
I20260928 06:48:06.331576 22845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:33141 every 8 connection(s)
I20260928 06:48:06.332289 22778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:48:06.334888 22846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:48:06.340054 22846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f: Bootstrap starting.
I20260928 06:48:06.342350 22846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f: Neither blocks nor log segments found. Creating new log.
I20260928 06:48:06.342535 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 22778
I20260928 06:48:06.342653 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:48:06.342921 7360 external_mini_cluster.cc:1428] Setting key 6c22dea2fda7b16552803bd1e3c4ea5c
I20260928 06:48:06.343138 22846 log.cc:824] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:48:06.345067 22846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f: No bootstrap required, opened a new log
I20260928 06:48:06.349114 22846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52938bc4827343ae836c286f671af74f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33141 } }
I20260928 06:48:06.349310 22846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:48:06.349417 22846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52938bc4827343ae836c286f671af74f, State: Initialized, Role: FOLLOWER
I20260928 06:48:06.349856 22846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52938bc4827343ae836c286f671af74f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33141 } }
I20260928 06:48:06.350049 22846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:48:06.350124 22846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:48:06.350221 22846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:48:06.351260 22846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52938bc4827343ae836c286f671af74f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33141 } }
Sep 28 06:48:06 dist-test-slave-twl6 krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:48:06.351600 22846 leader_election.cc:302] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52938bc4827343ae836c286f671af74f; no voters:
I20260928 06:48:06.351944 22846 leader_election.cc:288] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:48:06.352001 22851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:48:06.352221 22851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [term 1 LEADER]: Becoming Leader. State: Replica: 52938bc4827343ae836c286f671af74f, State: Running, Role: LEADER
I20260928 06:48:06.352553 22851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52938bc4827343ae836c286f671af74f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33141 } }
I20260928 06:48:06.353137 22846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:48:06.353710 22852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52938bc4827343ae836c286f671af74f. Latest consensus state: current_term: 1 leader_uuid: "52938bc4827343ae836c286f671af74f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52938bc4827343ae836c286f671af74f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33141 } } }
I20260928 06:48:06.353828 22852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [sys.catalog]: This master's current role is: LEADER
I20260928 06:48:06.354183 22853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52938bc4827343ae836c286f671af74f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52938bc4827343ae836c286f671af74f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33141 } } }
I20260928 06:48:06.354282 22853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [sys.catalog]: This master's current role is: LEADER
I20260928 06:48:06.355544 22857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:48:06.357487 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:06.344057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44260 (local address 127.7.48.62:33141)
0928 06:48:06.344426 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0928 06:48:06.344444 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 06:48:06.344491 (+ 47us) server_negotiation.cc:407] Connection header received
0928 06:48:06.345083 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:48:06.345099 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:48:06.345451 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:48:06.345726 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:48:06.346538 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:06.347484 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:06.348804 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:06.349262 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:06.349457 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:06.352180 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:48:06.352185 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:48:06.352196 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:48:06.352245 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:48:06.354262 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:06.354793 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:06.354802 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:06.354810 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:06.354971 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:06.355329 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:06.355334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:06.355339 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:06.355877 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:48:06.356125 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 06:48:06.356500 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 06:48:06.356810 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":105,"threads_started":1}
I20260928 06:48:06.359692 22857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:48:06.364866 22857 catalog_manager.cc:1347] Generated new cluster ID: d0ef55f99ef248aea06909b89a18ce86
I20260928 06:48:06.364955 22857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:48:06.377941 22857 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:48:06.379086 22857 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:48:06.385937 22857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f: Generated new TSK 0
I20260928 06:48:06.386551 22857 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
I20260928 06:48:06.419348 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:33141
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:48:06.539755 22872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:48:06.540058 22872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:48:06.540135 22872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:48:06.543799 22872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:48:06.543846 22872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:48:06.543964 22872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:48:06.549010 22872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:48:06.549974 22872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:48:06.550913 22872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:48:06.560468 22880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:48:06.560537 22878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:48:06.560501 22877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:48:06.560694 22872 server_base.cc:1034] running on GCE node
I20260928 06:48:06.561118 22872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:48:06.561764 22872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:48:06.562947 22872 hybrid_clock.cc:648] HybridClock initialized: now 1790578086562934 us; error 35 us; skew 500 ppm
Sep 28 06:48:06 dist-test-slave-twl6 krb5kdc[22765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578086, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:48:06.565611 22872 init.cc:377] Logged in from keytab as kudu/127.7.48.1@KRBTEST.COM (short username kudu)
I20260928 06:48:06.566421 22872 webserver.cc:460] Webserver started at http://127.7.48.1:34203/ using document root <none> and password file <none>
I20260928 06:48:06.566988 22872 fs_manager.cc:359] Metadata directory not provided
I20260928 06:48:06.567083 22872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:48:06.567317 22872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:48:06.569294 22872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "054e90aa690748eca63669817f350244"
format_stamp: "Formatted at 2026-09-28 06:48:06 on dist-test-slave-twl6"
server_key: "d07d5b40d0508bad8a1bc4b81119fc04"
server_key_iv: "1e4af7bbd3177d44050d936216ca6fa3"
server_key_version: "encryptionkey@0"
I20260928 06:48:06.569849 22872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "054e90aa690748eca63669817f350244"
format_stamp: "Formatted at 2026-09-28 06:48:06 on dist-test-slave-twl6"
server_key: "d07d5b40d0508bad8a1bc4b81119fc04"
server_key_iv: "1e4af7bbd3177d44050d936216ca6fa3"
server_key_version: "encryptionkey@0"
I20260928 06:48:06.573247 22872 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:48:06.575490 22887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:06.576817 22872 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:48:06.576951 22872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "054e90aa690748eca63669817f350244"
format_stamp: "Formatted at 2026-09-28 06:48:06 on dist-test-slave-twl6"
server_key: "d07d5b40d0508bad8a1bc4b81119fc04"
server_key_iv: "1e4af7bbd3177d44050d936216ca6fa3"
server_key_version: "encryptionkey@0"
I20260928 06:48:06.577067 22872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:48:06.586477 22872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:48:06.589563 22872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:48:06.589761 22872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:48:06.590301 22872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:48:06.591040 22872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:48:06.591114 22872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:06.591182 22872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:48:06.591224 22872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:06.599614 22872 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:40317
I20260928 06:48:06.599638 23000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:40317 every 8 connection(s)
I20260928 06:48:06.600471 22872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:48:06.605499 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 22872
I20260928 06:48:06.605628 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
I20260928 06:48:06.605897 7360 external_mini_cluster.cc:1428] Setting key fa57716afa7aa187a031ee923b33d62e
Sep 28 06:48:06 dist-test-slave-twl6 krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578086, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:48:06.614517 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:06.602074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:56665 (local address 127.7.48.62:33141)
0928 06:48:06.602209 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 06:48:06.602213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:48:06.602965 (+ 752us) server_negotiation.cc:407] Connection header received
0928 06:48:06.604411 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:48:06.604415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:48:06.604474 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:48:06.604584 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:48:06.605602 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:06.606213 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:06.607632 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:06.608112 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:06.608238 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:06.610674 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:48:06.610677 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:48:06.610678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:48:06.610702 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:48:06.612307 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:06.612947 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:06.612949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:06.612951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:06.613036 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:06.613445 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:06.613447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:06.613448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:06.613633 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:48:06.613755 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 06:48:06.614220 (+ 465us) server_negotiation.cc:299] Negotiation successful
0928 06:48:06.614346 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 06:48:06.615329 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:06.602410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.1:56665)
0928 06:48:06.602829 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:06.602867 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:48:06.604150 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:48:06.604762 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:48:06.604768 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:48:06.605068 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:48:06.605467 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:06.605479 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:06.606356 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:06.606378 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:06.607509 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:06.607519 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:06.608278 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:06.608282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:06.608415 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:06.609006 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:48:06.609022 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:48:06.610496 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:48:06.612430 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:48:06.612439 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:48:06.612456 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:48:06.612842 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:48:06.613135 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:48:06.613141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:48:06.613145 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:48:06.613338 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:48:06.613754 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:48:06.613761 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:48:06.613989 (+ 228us) client_negotiation.cc:769] Sending connection context
0928 06:48:06.614175 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 06:48:06.614475 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for kudu/127.7.48.2@KRBTEST.COM; defaulting to no policy
I20260928 06:48:06.616593 23001 heartbeater.cc:347] Connected to a master server at 127.7.48.62:33141
I20260928 06:48:06.616848 23001 heartbeater.cc:464] Registering TS with master...
I20260928 06:48:06.617306 23001 heartbeater.cc:511] Master 127.7.48.62:33141 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.2@KRBTEST.COM" created.
I20260928 06:48:06.618856 22810 ts_manager.cc:194] Registered new tserver with Master: 054e90aa690748eca63669817f350244 (127.7.48.1:40317)
I20260928 06:48:06.620414 22810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.1@KRBTEST.COM'} at 127.7.48.1:56665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
I20260928 06:48:06.634999 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:33141
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:48:06.739910 23006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:48:06.740177 23006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:48:06.740216 23006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:48:06.743862 23006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:48:06.743924 23006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:48:06.743995 23006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:48:06.748764 23006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:48:06.749711 23006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:48:06.750595 23006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:48:06.759847 23014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:48:06.759868 23012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:48:06.760017 23006 server_base.cc:1034] running on GCE node
W20260928 06:48:06.759860 23011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:48:06.760515 23006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:48:06.761133 23006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:48:06.762316 23006 hybrid_clock.cc:648] HybridClock initialized: now 1790578086762304 us; error 25 us; skew 500 ppm
Sep 28 06:48:06 dist-test-slave-twl6 krb5kdc[22765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578086, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:48:06.764963 23006 init.cc:377] Logged in from keytab as kudu/127.7.48.2@KRBTEST.COM (short username kudu)
I20260928 06:48:06.765749 23006 webserver.cc:460] Webserver started at http://127.7.48.2:42515/ using document root <none> and password file <none>
I20260928 06:48:06.766347 23006 fs_manager.cc:359] Metadata directory not provided
I20260928 06:48:06.766439 23006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:48:06.766671 23006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:48:06.768659 23006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "20932349d3e147af931d4d553f6005db"
format_stamp: "Formatted at 2026-09-28 06:48:06 on dist-test-slave-twl6"
server_key: "e8fbbc455c54bf8904e3935099b89268"
server_key_iv: "f49044de2e589fef6ae122a25c1dd437"
server_key_version: "encryptionkey@0"
I20260928 06:48:06.769251 23006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "20932349d3e147af931d4d553f6005db"
format_stamp: "Formatted at 2026-09-28 06:48:06 on dist-test-slave-twl6"
server_key: "e8fbbc455c54bf8904e3935099b89268"
server_key_iv: "f49044de2e589fef6ae122a25c1dd437"
server_key_version: "encryptionkey@0"
I20260928 06:48:06.772787 23006 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:48:06.775084 23021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:06.776255 23006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 06:48:06.776382 23006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "20932349d3e147af931d4d553f6005db"
format_stamp: "Formatted at 2026-09-28 06:48:06 on dist-test-slave-twl6"
server_key: "e8fbbc455c54bf8904e3935099b89268"
server_key_iv: "f49044de2e589fef6ae122a25c1dd437"
server_key_version: "encryptionkey@0"
I20260928 06:48:06.776489 23006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:48:06.797300 23006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:48:06.800427 23006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:48:06.800611 23006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:48:06.801138 23006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:48:06.801918 23006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:48:06.801983 23006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:06.802047 23006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:48:06.802111 23006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:06.811080 23006 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:45701
I20260928 06:48:06.811142 23134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:45701 every 8 connection(s)
I20260928 06:48:06.812011 23006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:48:06.820396 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 23006
I20260928 06:48:06.820497 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:48:06.820798 7360 external_mini_cluster.cc:1428] Setting key c2d1966f767e95a32ec9b97ab392b842
Sep 28 06:48:06 dist-test-slave-twl6 krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578086, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:48:06.824944 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:06.813748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:58959 (local address 127.7.48.62:33141)
0928 06:48:06.813884 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 06:48:06.813888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:48:06.814711 (+ 823us) server_negotiation.cc:407] Connection header received
0928 06:48:06.815918 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:48:06.815921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:48:06.815977 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:48:06.816054 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:48:06.817147 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:06.817680 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:06.818796 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:06.819019 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:06.819159 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:06.821598 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:48:06.821601 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:48:06.821603 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:48:06.821626 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:48:06.822997 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:06.823482 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:06.823486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:06.823487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:06.823568 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:06.823919 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:06.823923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:06.823924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:06.824094 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:48:06.824211 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 06:48:06.824666 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 06:48:06.824781 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:48:06.825721 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:06.814109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.2:58959)
0928 06:48:06.814535 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:06.814576 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:48:06.815709 (+ 1133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:48:06.816209 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:48:06.816215 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:48:06.816533 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:48:06.817001 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:06.817013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:06.817809 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:06.817818 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:06.818670 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:06.818679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:06.819186 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:06.819190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:06.819373 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:06.819996 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:48:06.820012 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:48:06.821414 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:48:06.823117 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:48:06.823123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:48:06.823135 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:48:06.823379 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:48:06.823667 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:48:06.823670 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:48:06.823672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:48:06.823804 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:48:06.824209 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:48:06.824216 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:48:06.824442 (+ 226us) client_negotiation.cc:769] Sending connection context
0928 06:48:06.824620 (+ 178us) client_negotiation.cc:240] Negotiation successful
0928 06:48:06.824917 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":118,"threads_started":1}
I20260928 06:48:06.826840 23135 heartbeater.cc:347] Connected to a master server at 127.7.48.62:33141
I20260928 06:48:06.827104 23135 heartbeater.cc:464] Registering TS with master...
I20260928 06:48:06.827699 23135 heartbeater.cc:511] Master 127.7.48.62:33141 requested a full tablet report, sending...
I20260928 06:48:06.828867 22810 ts_manager.cc:194] Registered new tserver with Master: 20932349d3e147af931d4d553f6005db (127.7.48.2:45701)
I20260928 06:48:06.829681 22810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.2@KRBTEST.COM'} at 127.7.48.2:58959
WARNING: no policy specified for kudu/127.7.48.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
I20260928 06:48:06.846851 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:33141
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:48:06.953609 23140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:48:06.953835 23140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:48:06.953874 23140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:48:06.958257 23140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:48:06.958308 23140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:48:06.958359 23140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:48:06.963039 23140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:48:06.963946 23140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:48:06.964814 23140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:48:06.973690 23148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:48:06.973748 23145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:48:06.973830 23146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:48:06.973727 23140 server_base.cc:1034] running on GCE node
I20260928 06:48:06.974260 23140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:48:06.974817 23140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:48:06.975952 23140 hybrid_clock.cc:648] HybridClock initialized: now 1790578086975921 us; error 37 us; skew 500 ppm
Sep 28 06:48:06 dist-test-slave-twl6 krb5kdc[22765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578086, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:48:06.978528 23140 init.cc:377] Logged in from keytab as kudu/127.7.48.3@KRBTEST.COM (short username kudu)
I20260928 06:48:06.979318 23140 webserver.cc:460] Webserver started at http://127.7.48.3:35789/ using document root <none> and password file <none>
I20260928 06:48:06.979982 23140 fs_manager.cc:359] Metadata directory not provided
I20260928 06:48:06.980044 23140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:48:06.980304 23140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:48:06.982324 23140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "b5c2c05c3b944b8b9d1f8889988ad296"
format_stamp: "Formatted at 2026-09-28 06:48:06 on dist-test-slave-twl6"
server_key: "6ec4d015935492f95f28443d53b0381e"
server_key_iv: "9b87daf790ba59bde6c5199748b94b01"
server_key_version: "encryptionkey@0"
I20260928 06:48:06.982787 23140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "b5c2c05c3b944b8b9d1f8889988ad296"
format_stamp: "Formatted at 2026-09-28 06:48:06 on dist-test-slave-twl6"
server_key: "6ec4d015935492f95f28443d53b0381e"
server_key_iv: "9b87daf790ba59bde6c5199748b94b01"
server_key_version: "encryptionkey@0"
I20260928 06:48:06.986127 23140 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 06:48:06.988354 23155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:06.989424 23140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:48:06.989511 23140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "b5c2c05c3b944b8b9d1f8889988ad296"
format_stamp: "Formatted at 2026-09-28 06:48:06 on dist-test-slave-twl6"
server_key: "6ec4d015935492f95f28443d53b0381e"
server_key_iv: "9b87daf790ba59bde6c5199748b94b01"
server_key_version: "encryptionkey@0"
I20260928 06:48:06.989586 23140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:48:07.004657 23140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:48:07.007249 23140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:48:07.007405 23140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:48:07.008031 23140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:48:07.008737 23140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:48:07.008780 23140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:07.008814 23140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:48:07.008834 23140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:07.017673 23140 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:43811
I20260928 06:48:07.017697 23268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:43811 every 8 connection(s)
I20260928 06:48:07.018471 23140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:48:07.022075 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 23140
I20260928 06:48:07.022198 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:48:07.022459 7360 external_mini_cluster.cc:1428] Setting key 44eefa3fb97eb8d375026e17799a1234
Sep 28 06:48:07 dist-test-slave-twl6 krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578086, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:48:07.031508 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:07.020138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:44231 (local address 127.7.48.62:33141)
0928 06:48:07.020308 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 06:48:07.020312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:48:07.021526 (+ 1214us) server_negotiation.cc:407] Connection header received
0928 06:48:07.022395 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:48:07.022398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:48:07.022453 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:48:07.022533 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:48:07.023933 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.024490 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.025647 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.025855 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.025952 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:07.028256 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:48:07.028259 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:48:07.028262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:48:07.028291 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:48:07.029501 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:07.030008 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:07.030012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:07.030014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:07.030099 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:07.030456 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:07.030459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:07.030461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:07.030634 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:48:07.030741 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 06:48:07.031233 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 06:48:07.031355 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 06:48:07.032284 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:07.020571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.3:44231)
0928 06:48:07.021383 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:07.021421 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:48:07.022147 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:48:07.022704 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:48:07.022710 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:48:07.023117 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:48:07.023760 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:07.023784 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.024620 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.024624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:07.025519 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:07.025528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.025995 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.025999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:07.026163 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:07.026656 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:48:07.026670 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:48:07.028112 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:48:07.029621 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:48:07.029627 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:48:07.029640 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:48:07.029903 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:48:07.030221 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:48:07.030225 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:48:07.030227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:48:07.030360 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:48:07.030740 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:48:07.030746 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:48:07.030978 (+ 232us) client_negotiation.cc:769] Sending connection context
0928 06:48:07.031171 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 06:48:07.031462 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":514,"threads_started":1}
I20260928 06:48:07.033277 23269 heartbeater.cc:347] Connected to a master server at 127.7.48.62:33141
I20260928 06:48:07.033521 23269 heartbeater.cc:464] Registering TS with master...
I20260928 06:48:07.033964 23269 heartbeater.cc:511] Master 127.7.48.62:33141 requested a full tablet report, sending...
I20260928 06:48:07.034931 22810 ts_manager.cc:194] Registered new tserver with Master: b5c2c05c3b944b8b9d1f8889988ad296 (127.7.48.3:43811)
I20260928 06:48:07.035550 22810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.3@KRBTEST.COM'} at 127.7.48.3:44231
I20260928 06:48:07.036280 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:48:07.044998 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:07.038279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44264 (local address 127.7.48.62:33141)
0928 06:48:07.038394 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 06:48:07.038397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:48:07.038545 (+ 148us) server_negotiation.cc:407] Connection header received
0928 06:48:07.038639 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:48:07.038641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:48:07.038676 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:48:07.038745 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:48:07.039352 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.039835 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.040691 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.040901 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.040989 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:07.042315 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:48:07.042318 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:48:07.042320 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:48:07.042342 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:48:07.043561 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:07.043974 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:07.043977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:07.043978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:07.044053 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:07.044288 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:07.044290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:07.044291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:07.044491 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:48:07.044598 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 06:48:07.044802 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 06:48:07.044889 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:48:07.047850 22809 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:48:07.050848 22809 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:48:07.064680 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:07.059301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:43811 (local address 127.0.0.1:56358)
0928 06:48:07.059938 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:07.059952 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:48:07.060034 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:48:07.060718 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:48:07.060721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:48:07.060738 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:48:07.060840 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:07.060848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.061850 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.061853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:07.063042 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:07.063052 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.064130 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.064134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:07.064247 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:07.064284 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 06:48:07.064421 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 06:48:07.064544 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
I20260928 06:48:07.064692 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:07.059729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56358 (local address 127.7.48.3:43811)
0928 06:48:07.060254 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0928 06:48:07.060268 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:48:07.060289 (+ 21us) server_negotiation.cc:407] Connection header received
0928 06:48:07.060361 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:48:07.060367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:48:07.060578 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:48:07.060737 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:48:07.060963 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.061715 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.063184 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.063999 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.064128 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:07.064272 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 06:48:07.064452 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 06:48:07.064542 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":147,"threads_started":1}
I20260928 06:48:07.064976 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:07.058942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:45701 (local address 127.0.0.1:43706)
0928 06:48:07.059299 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:07.059350 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:48:07.059480 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:48:07.060127 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:48:07.060133 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:48:07.060157 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:48:07.060430 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:07.060441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.062302 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.062306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:07.063489 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:07.063498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.064580 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.064586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:07.064686 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:07.064707 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 06:48:07.064756 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:48:07.064842 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":74,"thread_start_us":89,"threads_started":1}
I20260928 06:48:07.065294 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:07.059643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:40317 (local address 127.0.0.1:35686)
0928 06:48:07.060202 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:07.060213 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:48:07.060306 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:48:07.060897 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:48:07.060901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:48:07.060919 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:48:07.061020 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:07.061025 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.062539 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.062542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:07.063645 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:07.063654 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.064905 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.064909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:07.065004 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:07.065030 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 06:48:07.065095 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:48:07.065172 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":54,"thread_start_us":91,"threads_started":1}
I20260928 06:48:07.065755 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:07.059162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43706 (local address 127.7.48.2:45701)
0928 06:48:07.059613 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0928 06:48:07.059626 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:48:07.059651 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:48:07.059731 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:48:07.059736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:48:07.059945 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:48:07.060970 (+ 1025us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:48:07.061120 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.062160 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.063619 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.064433 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.065248 (+ 815us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:07.065494 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 06:48:07.065535 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 06:48:07.065616 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":141,"threads_started":1}
I20260928 06:48:07.065755 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:07.059902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35686 (local address 127.7.48.1:40317)
0928 06:48:07.060489 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0928 06:48:07.060502 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:48:07.060524 (+ 22us) server_negotiation.cc:407] Connection header received
0928 06:48:07.060608 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:48:07.060613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:48:07.060777 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:48:07.060910 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:48:07.061133 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.062167 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.063765 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.064542 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.064645 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:07.065480 (+ 835us) server_negotiation.cc:1035] Waiting for connection context
0928 06:48:07.065532 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 06:48:07.065627 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":148,"threads_started":1}
I20260928 06:48:07.067982 23203 tablet_service.cc:1467] Processing CreateTablet for tablet fabfc261ae2f4de29eb108c47f5211ac (DEFAULT_TABLE table=test-table [id=d421815d84334360af103c3eb97c2339]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:48:07.068486 22935 tablet_service.cc:1467] Processing CreateTablet for tablet fabfc261ae2f4de29eb108c47f5211ac (DEFAULT_TABLE table=test-table [id=d421815d84334360af103c3eb97c2339]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:48:07.068666 23069 tablet_service.cc:1467] Processing CreateTablet for tablet fabfc261ae2f4de29eb108c47f5211ac (DEFAULT_TABLE table=test-table [id=d421815d84334360af103c3eb97c2339]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:48:07.069093 23203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fabfc261ae2f4de29eb108c47f5211ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:48:07.069440 22935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fabfc261ae2f4de29eb108c47f5211ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:48:07.069509 23069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fabfc261ae2f4de29eb108c47f5211ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:48:07.075074 23286 tablet_bootstrap.cc:492] T fabfc261ae2f4de29eb108c47f5211ac P 20932349d3e147af931d4d553f6005db: Bootstrap starting.
I20260928 06:48:07.076679 23287 tablet_bootstrap.cc:492] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244: Bootstrap starting.
I20260928 06:48:07.076781 23288 tablet_bootstrap.cc:492] T fabfc261ae2f4de29eb108c47f5211ac P b5c2c05c3b944b8b9d1f8889988ad296: Bootstrap starting.
I20260928 06:48:07.078238 23286 tablet_bootstrap.cc:654] T fabfc261ae2f4de29eb108c47f5211ac P 20932349d3e147af931d4d553f6005db: Neither blocks nor log segments found. Creating new log.
I20260928 06:48:07.078764 23288 tablet_bootstrap.cc:654] T fabfc261ae2f4de29eb108c47f5211ac P b5c2c05c3b944b8b9d1f8889988ad296: Neither blocks nor log segments found. Creating new log.
I20260928 06:48:07.078765 23287 tablet_bootstrap.cc:654] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244: Neither blocks nor log segments found. Creating new log.
I20260928 06:48:07.078962 23286 log.cc:824] T fabfc261ae2f4de29eb108c47f5211ac P 20932349d3e147af931d4d553f6005db: Log is configured to *not* fsync() on all Append() calls
I20260928 06:48:07.080036 23287 log.cc:824] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244: Log is configured to *not* fsync() on all Append() calls
I20260928 06:48:07.080176 23288 log.cc:824] T fabfc261ae2f4de29eb108c47f5211ac P b5c2c05c3b944b8b9d1f8889988ad296: Log is configured to *not* fsync() on all Append() calls
I20260928 06:48:07.081080 23286 tablet_bootstrap.cc:492] T fabfc261ae2f4de29eb108c47f5211ac P 20932349d3e147af931d4d553f6005db: No bootstrap required, opened a new log
I20260928 06:48:07.081295 23286 ts_tablet_manager.cc:1397] T fabfc261ae2f4de29eb108c47f5211ac P 20932349d3e147af931d4d553f6005db: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 06:48:07.081950 23287 tablet_bootstrap.cc:492] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244: No bootstrap required, opened a new log
I20260928 06:48:07.082113 23287 ts_tablet_manager.cc:1397] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 06:48:07.082149 23288 tablet_bootstrap.cc:492] T fabfc261ae2f4de29eb108c47f5211ac P b5c2c05c3b944b8b9d1f8889988ad296: No bootstrap required, opened a new log
I20260928 06:48:07.082317 23288 ts_tablet_manager.cc:1397] T fabfc261ae2f4de29eb108c47f5211ac P b5c2c05c3b944b8b9d1f8889988ad296: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260928 06:48:07.084946 23288 raft_consensus.cc:348] T fabfc261ae2f4de29eb108c47f5211ac P b5c2c05c3b944b8b9d1f8889988ad296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } }
I20260928 06:48:07.084928 23286 raft_consensus.cc:348] T fabfc261ae2f4de29eb108c47f5211ac P 20932349d3e147af931d4d553f6005db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } }
I20260928 06:48:07.085175 23288 raft_consensus.cc:374] T fabfc261ae2f4de29eb108c47f5211ac P b5c2c05c3b944b8b9d1f8889988ad296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:48:07.085175 23286 raft_consensus.cc:374] T fabfc261ae2f4de29eb108c47f5211ac P 20932349d3e147af931d4d553f6005db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:48:07.085224 23288 raft_consensus.cc:729] T fabfc261ae2f4de29eb108c47f5211ac P b5c2c05c3b944b8b9d1f8889988ad296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c2c05c3b944b8b9d1f8889988ad296, State: Initialized, Role: FOLLOWER
I20260928 06:48:07.085232 23286 raft_consensus.cc:729] T fabfc261ae2f4de29eb108c47f5211ac P 20932349d3e147af931d4d553f6005db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20932349d3e147af931d4d553f6005db, State: Initialized, Role: FOLLOWER
I20260928 06:48:07.085628 23288 consensus_queue.cc:260] T fabfc261ae2f4de29eb108c47f5211ac P b5c2c05c3b944b8b9d1f8889988ad296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } }
I20260928 06:48:07.085642 23286 consensus_queue.cc:260] T fabfc261ae2f4de29eb108c47f5211ac P 20932349d3e147af931d4d553f6005db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } }
I20260928 06:48:07.085604 23287 raft_consensus.cc:348] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } }
I20260928 06:48:07.085803 23287 raft_consensus.cc:374] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:48:07.085862 23287 raft_consensus.cc:729] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 054e90aa690748eca63669817f350244, State: Initialized, Role: FOLLOWER
I20260928 06:48:07.086246 23287 consensus_queue.cc:260] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } }
I20260928 06:48:07.086541 23286 ts_tablet_manager.cc:1428] T fabfc261ae2f4de29eb108c47f5211ac P 20932349d3e147af931d4d553f6005db: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 06:48:07.086555 23288 ts_tablet_manager.cc:1428] T fabfc261ae2f4de29eb108c47f5211ac P b5c2c05c3b944b8b9d1f8889988ad296: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260928 06:48:07.086887 23135 heartbeater.cc:503] Master 127.7.48.62:33141 was elected leader, sending a full tablet report...
I20260928 06:48:07.087056 23287 ts_tablet_manager.cc:1428] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 06:48:07.087117 23269 heartbeater.cc:503] Master 127.7.48.62:33141 was elected leader, sending a full tablet report...
I20260928 06:48:07.087239 23001 heartbeater.cc:503] Master 127.7.48.62:33141 was elected leader, sending a full tablet report...
W20260928 06:48:07.101743 23002 tablet.cc:2369] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:48:07.269416 23270 tablet.cc:2369] T fabfc261ae2f4de29eb108c47f5211ac P b5c2c05c3b944b8b9d1f8889988ad296: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:48:07.312016 23294 raft_consensus.cc:482] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:48:07.312247 23294 raft_consensus.cc:504] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } }
W20260928 06:48:07.313077 23136 tablet.cc:2369] T fabfc261ae2f4de29eb108c47f5211ac P 20932349d3e147af931d4d553f6005db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:48:07.313405 23294 leader_election.cc:288] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20932349d3e147af931d4d553f6005db (127.7.48.2:45701), b5c2c05c3b944b8b9d1f8889988ad296 (127.7.48.3:43811)
I20260928 06:48:07.318717 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:07.314085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:57911 (local address 127.7.48.2:45701)
0928 06:48:07.314220 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 06:48:07.314223 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:48:07.314236 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:48:07.314308 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:48:07.314311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:48:07.314363 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:48:07.314457 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:48:07.314909 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.315624 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.317377 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.318013 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.318123 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:07.318189 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:48:07.318525 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 06:48:07.318591 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:48:07.318717 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:07.313843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:35813 (local address 127.7.48.3:43811)
0928 06:48:07.313973 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 06:48:07.313980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:48:07.314161 (+ 181us) server_negotiation.cc:407] Connection header received
0928 06:48:07.314329 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:48:07.314332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:48:07.314386 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:48:07.314463 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:48:07.314909 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.315624 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.317377 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.318014 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.318123 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:07.318189 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:48:07.318525 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 06:48:07.318591 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:48:07.318766 23296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:07.313693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:43811 (local address 127.7.48.1:35813)
0928 06:48:07.314055 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:07.314081 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 06:48:07.314220 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:48:07.314491 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:48:07.314495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:48:07.314536 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:48:07.314782 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:07.314789 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.315733 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.315736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:07.317239 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:07.317252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.318146 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.318150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:07.318369 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:07.318408 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 06:48:07.318496 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:48:07.318599 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":262,"spinlock_wait_cycles":7936,"thread_start_us":123,"threads_started":1}
I20260928 06:48:07.318884 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:07.313660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:45701 (local address 127.7.48.1:57911)
0928 06:48:07.313922 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:07.313941 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 06:48:07.314047 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:48:07.314491 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:48:07.314494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:48:07.314535 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:48:07.314785 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:07.314789 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.315733 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.315736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:07.317246 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:07.317252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.318146 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.318150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:07.318365 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:07.318419 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 06:48:07.318496 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 06:48:07.318599 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":335,"spinlock_wait_cycles":29952,"thread_start_us":95,"threads_started":1}
I20260928 06:48:07.319555 23223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabfc261ae2f4de29eb108c47f5211ac" candidate_uuid: "054e90aa690748eca63669817f350244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" is_pre_election: true
I20260928 06:48:07.319548 23089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabfc261ae2f4de29eb108c47f5211ac" candidate_uuid: "054e90aa690748eca63669817f350244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20932349d3e147af931d4d553f6005db" is_pre_election: true
I20260928 06:48:07.319861 23223 raft_consensus.cc:2454] T fabfc261ae2f4de29eb108c47f5211ac P b5c2c05c3b944b8b9d1f8889988ad296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 054e90aa690748eca63669817f350244 in term 0.
I20260928 06:48:07.319918 23089 raft_consensus.cc:2454] T fabfc261ae2f4de29eb108c47f5211ac P 20932349d3e147af931d4d553f6005db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 054e90aa690748eca63669817f350244 in term 0.
I20260928 06:48:07.320430 22889 leader_election.cc:302] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 054e90aa690748eca63669817f350244, 20932349d3e147af931d4d553f6005db, b5c2c05c3b944b8b9d1f8889988ad296; no voters:
I20260928 06:48:07.320704 23294 raft_consensus.cc:2781] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:48:07.320775 23294 raft_consensus.cc:482] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:48:07.320820 23294 raft_consensus.cc:3037] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:48:07.321833 23294 raft_consensus.cc:504] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } }
I20260928 06:48:07.322162 23294 leader_election.cc:288] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244 [CANDIDATE]: Term 1 election: Requested vote from peers 20932349d3e147af931d4d553f6005db (127.7.48.2:45701), b5c2c05c3b944b8b9d1f8889988ad296 (127.7.48.3:43811)
I20260928 06:48:07.322486 23089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabfc261ae2f4de29eb108c47f5211ac" candidate_uuid: "054e90aa690748eca63669817f350244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20932349d3e147af931d4d553f6005db"
I20260928 06:48:07.322495 23223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabfc261ae2f4de29eb108c47f5211ac" candidate_uuid: "054e90aa690748eca63669817f350244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c2c05c3b944b8b9d1f8889988ad296"
I20260928 06:48:07.322609 23089 raft_consensus.cc:3037] T fabfc261ae2f4de29eb108c47f5211ac P 20932349d3e147af931d4d553f6005db [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:48:07.322615 23223 raft_consensus.cc:3037] T fabfc261ae2f4de29eb108c47f5211ac P b5c2c05c3b944b8b9d1f8889988ad296 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:48:07.323596 23089 raft_consensus.cc:2454] T fabfc261ae2f4de29eb108c47f5211ac P 20932349d3e147af931d4d553f6005db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 054e90aa690748eca63669817f350244 in term 1.
I20260928 06:48:07.323659 23223 raft_consensus.cc:2454] T fabfc261ae2f4de29eb108c47f5211ac P b5c2c05c3b944b8b9d1f8889988ad296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 054e90aa690748eca63669817f350244 in term 1.
I20260928 06:48:07.323946 22888 leader_election.cc:302] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 054e90aa690748eca63669817f350244, 20932349d3e147af931d4d553f6005db; no voters:
I20260928 06:48:07.324102 23294 raft_consensus.cc:2781] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:48:07.324308 23294 raft_consensus.cc:686] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244 [term 1 LEADER]: Becoming Leader. State: Replica: 054e90aa690748eca63669817f350244, State: Running, Role: LEADER
I20260928 06:48:07.324621 23294 consensus_queue.cc:237] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } }
I20260928 06:48:07.327562 22810 catalog_manager.cc:5495] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244 reported cstate change: term changed from 0 to 1, leader changed from <none> to 054e90aa690748eca63669817f350244 (127.7.48.1). New cstate: current_term: 1 leader_uuid: "054e90aa690748eca63669817f350244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } health_report { overall_health: UNKNOWN } } }
I20260928 06:48:07.361090 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:07.358015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35700 (local address 127.7.48.1:40317)
0928 06:48:07.358160 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 06:48:07.358163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:48:07.358191 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:48:07.358262 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:48:07.358264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:48:07.358318 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:48:07.358398 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:48:07.358600 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.359072 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.359858 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:07.360121 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:07.360206 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:07.360444 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:48:07.360812 (+ 368us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:48:07.360894 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:48:07.360946 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 06:48:07.360989 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 06:48:07.366844 23223 raft_consensus.cc:1261] T fabfc261ae2f4de29eb108c47f5211ac P b5c2c05c3b944b8b9d1f8889988ad296 [term 1 FOLLOWER]: Refusing update from remote peer 054e90aa690748eca63669817f350244: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:48:07.366844 23089 raft_consensus.cc:1261] T fabfc261ae2f4de29eb108c47f5211ac P 20932349d3e147af931d4d553f6005db [term 1 FOLLOWER]: Refusing update from remote peer 054e90aa690748eca63669817f350244: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:48:07.367578 23297 consensus_queue.cc:1035] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:48:07.367784 23294 consensus_queue.cc:1035] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:48:07.375067 23301 mvcc.cc:205] Tried to move back new op lower bound from 7334207845847040000 to 7334207845683957760. Current Snapshot: MvccSnapshot[applied={T|T < 7334207845847040000}]
I20260928 06:48:07.375674 23302 mvcc.cc:205] Tried to move back new op lower bound from 7334207845847040000 to 7334207845683957760. Current Snapshot: MvccSnapshot[applied={T|T < 7334207845847040000}]
I20260928 06:48:07.383584 22810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:48:07.383764 22810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:48:07.386103 22810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f: Sending DeleteTablet for 3 replicas of tablet fabfc261ae2f4de29eb108c47f5211ac
I20260928 06:48:07.386876 23069 tablet_service.cc:1514] Processing DeleteTablet for tablet fabfc261ae2f4de29eb108c47f5211ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:48:07 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:43706
I20260928 06:48:07.386956 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 22872
I20260928 06:48:07.386935 23203 tablet_service.cc:1514] Processing DeleteTablet for tablet fabfc261ae2f4de29eb108c47f5211ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:48:07 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:56358
I20260928 06:48:07.387766 23310 tablet_replica.cc:330] T fabfc261ae2f4de29eb108c47f5211ac P 20932349d3e147af931d4d553f6005db: stopping tablet replica
I20260928 06:48:07.388125 23310 raft_consensus.cc:2229] T fabfc261ae2f4de29eb108c47f5211ac P 20932349d3e147af931d4d553f6005db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:48:07.388449 23310 raft_consensus.cc:2258] T fabfc261ae2f4de29eb108c47f5211ac P 20932349d3e147af931d4d553f6005db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:48:07.389072 23311 tablet_replica.cc:330] T fabfc261ae2f4de29eb108c47f5211ac P b5c2c05c3b944b8b9d1f8889988ad296: stopping tablet replica
I20260928 06:48:07.389343 23311 raft_consensus.cc:2229] T fabfc261ae2f4de29eb108c47f5211ac P b5c2c05c3b944b8b9d1f8889988ad296 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:48:07.389607 23311 raft_consensus.cc:2258] T fabfc261ae2f4de29eb108c47f5211ac P b5c2c05c3b944b8b9d1f8889988ad296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:48:07.389801 23310 ts_tablet_manager.cc:1905] T fabfc261ae2f4de29eb108c47f5211ac P 20932349d3e147af931d4d553f6005db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:48:07.390959 23311 ts_tablet_manager.cc:1905] T fabfc261ae2f4de29eb108c47f5211ac P b5c2c05c3b944b8b9d1f8889988ad296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:48:07.392465 23310 ts_tablet_manager.cc:1918] T fabfc261ae2f4de29eb108c47f5211ac P 20932349d3e147af931d4d553f6005db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:48:07.392560 23310 log.cc:1196] T fabfc261ae2f4de29eb108c47f5211ac P 20932349d3e147af931d4d553f6005db: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/fabfc261ae2f4de29eb108c47f5211ac
I20260928 06:48:07.392859 23310 ts_tablet_manager.cc:1939] T fabfc261ae2f4de29eb108c47f5211ac P 20932349d3e147af931d4d553f6005db: Deleting consensus metadata
I20260928 06:48:07.393585 23311 ts_tablet_manager.cc:1918] T fabfc261ae2f4de29eb108c47f5211ac P b5c2c05c3b944b8b9d1f8889988ad296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:48:07.393692 23311 log.cc:1196] T fabfc261ae2f4de29eb108c47f5211ac P b5c2c05c3b944b8b9d1f8889988ad296: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/fabfc261ae2f4de29eb108c47f5211ac
I20260928 06:48:07.394026 23311 ts_tablet_manager.cc:1939] T fabfc261ae2f4de29eb108c47f5211ac P b5c2c05c3b944b8b9d1f8889988ad296: Deleting consensus metadata
I20260928 06:48:07.394132 22794 catalog_manager.cc:4841] TS 20932349d3e147af931d4d553f6005db (127.7.48.2:45701): tablet fabfc261ae2f4de29eb108c47f5211ac (table test-table [id=d421815d84334360af103c3eb97c2339]) successfully deleted
W20260928 06:48:07.394325 22794 connection.cc:535] client connection to 127.7.48.1:40317 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 06:48:07.394454 22794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:48:07.394625 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 23006
I20260928 06:48:07.395617 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:07.395364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:40317 (local address 127.0.0.1:35704)
0928 06:48:07.395456 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:07.395523 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.1:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 06:48:07.395635 22795 catalog_manager.cc:4841] TS b5c2c05c3b944b8b9d1f8889988ad296 (127.7.48.3:43811): tablet fabfc261ae2f4de29eb108c47f5211ac (table test-table [id=d421815d84334360af103c3eb97c2339]) successfully deleted
W20260928 06:48:07.395826 22794 catalog_manager.cc:4584] TS 054e90aa690748eca63669817f350244 (127.7.48.1:40317): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fabfc261ae2f4de29eb108c47f5211ac: Network error: Client connection negotiation failed: client connection to 127.7.48.1:40317: connect: Connection refused (error 111)
I20260928 06:48:07.400393 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 23140
I20260928 06:48:07.405763 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 22778
I20260928 06:48:27.412146 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:33141
--webserver_interface=127.7.48.62
--webserver_port=41089
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:33141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:48:27.518241 23313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:48:27.518474 23313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:48:27.518543 23313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:48:27.522274 23313 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 06:48:27.522315 23313 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 06:48:27.522330 23313 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 06:48:27.522343 23313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:48:27.522359 23313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:48:27.522375 23313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:48:27.522390 23313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:48:27.522405 23313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:48:27.527014 23313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:33141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:33141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=41089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:48:27.527927 23313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:48:27.528898 23313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:48:27.541467 23320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:48:27.541440 23319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:48:27.541541 23313 server_base.cc:1034] running on GCE node
W20260928 06:48:27.541461 23322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:48:27.542336 23313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:48:27.543407 23313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:48:27.544575 23313 hybrid_clock.cc:648] HybridClock initialized: now 1790578107544558 us; error 28 us; skew 500 ppm
Sep 28 06:48:27 dist-test-slave-twl6 krb5kdc[22765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578107, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:48:27.551209 23313 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:48:27.552054 23313 webserver.cc:460] Webserver started at http://127.7.48.62:41089/ using document root <none> and password file <none>
I20260928 06:48:27.552562 23313 fs_manager.cc:359] Metadata directory not provided
I20260928 06:48:27.552627 23313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:48:27.556389 23313 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 06:48:27.558282 23329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:27.559478 23313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:48:27.559602 23313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "52938bc4827343ae836c286f671af74f"
format_stamp: "Formatted at 2026-09-28 06:48:06 on dist-test-slave-twl6"
server_key: "4608f488d78d9b4f78aa11fbc9eec076"
server_key_iv: "2a3e05411162a03641ae8209db572de0"
server_key_version: "encryptionkey@0"
I20260928 06:48:27.560237 23313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:48:27.574049 23313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:48:27.577157 23313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:48:27.577340 23313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:48:27.584323 23313 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:33141
I20260928 06:48:27.584612 23381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:33141 every 8 connection(s)
I20260928 06:48:27.585391 23313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:48:27.587949 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 23313
I20260928 06:48:27.588410 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:40317
--local_ip_for_outbound_sockets=127.7.48.1
--tserver_master_addrs=127.7.48.62:33141
--webserver_port=34203
--webserver_interface=127.7.48.1
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:48:27.590211 23382 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:48:27.591804 23382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f: Bootstrap starting.
I20260928 06:48:27.608517 23382 log.cc:824] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:48:27.616971 23382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:48:27.617326 23382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f: Bootstrap complete.
I20260928 06:48:27.621359 23382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52938bc4827343ae836c286f671af74f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33141 } }
I20260928 06:48:27.621714 23382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52938bc4827343ae836c286f671af74f, State: Initialized, Role: FOLLOWER
I20260928 06:48:27.622201 23382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52938bc4827343ae836c286f671af74f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33141 } }
I20260928 06:48:27.622308 23382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:48:27.622390 23382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:48:27.622496 23382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:48:27.624358 23382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52938bc4827343ae836c286f671af74f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33141 } }
I20260928 06:48:27.624673 23382 leader_election.cc:302] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52938bc4827343ae836c286f671af74f; no voters:
I20260928 06:48:27.625001 23382 leader_election.cc:288] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:48:27.625087 23386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:48:27.625300 23386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [term 2 LEADER]: Becoming Leader. State: Replica: 52938bc4827343ae836c286f671af74f, State: Running, Role: LEADER
I20260928 06:48:27.625612 23386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52938bc4827343ae836c286f671af74f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33141 } }
I20260928 06:48:27.626214 23382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:48:27.626989 23387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "52938bc4827343ae836c286f671af74f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52938bc4827343ae836c286f671af74f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33141 } } }
I20260928 06:48:27.627126 23387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [sys.catalog]: This master's current role is: LEADER
I20260928 06:48:27.627489 23388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52938bc4827343ae836c286f671af74f. Latest consensus state: current_term: 2 leader_uuid: "52938bc4827343ae836c286f671af74f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52938bc4827343ae836c286f671af74f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33141 } } }
I20260928 06:48:27.627619 23388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [sys.catalog]: This master's current role is: LEADER
I20260928 06:48:27.627905 23395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:48:27.632755 23395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:48:27.633560 23395 catalog_manager.cc:1259] Loaded cluster ID: d0ef55f99ef248aea06909b89a18ce86
I20260928 06:48:27.633615 23395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:48:27.635872 23395 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:48:27.636579 23395 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f: Loaded TSK: 0
I20260928 06:48:27.655802 23395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f: Generated new TSK 1
I20260928 06:48:27.656113 23395 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:48:27.703318 23384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:48:27.703616 23384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:48:27.703685 23384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:48:27.707288 23384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:48:27.707366 23384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:48:27.707453 23384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:48:27.712771 23384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:40317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=34203
--tserver_master_addrs=127.7.48.62:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:48:27.713745 23384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:48:27.714695 23384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:48:27.724556 23410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:48:27.724577 23409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:48:27.724576 23412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:48:27.725051 23384 server_base.cc:1034] running on GCE node
I20260928 06:48:27.725535 23384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:48:27.726356 23384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:48:27.727540 23384 hybrid_clock.cc:648] HybridClock initialized: now 1790578107727520 us; error 36 us; skew 500 ppm
Sep 28 06:48:27 dist-test-slave-twl6 krb5kdc[22765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578107, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:48:27.730829 23384 init.cc:377] Logged in from keytab as kudu/127.7.48.1@KRBTEST.COM (short username kudu)
I20260928 06:48:27.731717 23384 webserver.cc:460] Webserver started at http://127.7.48.1:34203/ using document root <none> and password file <none>
I20260928 06:48:27.732304 23384 fs_manager.cc:359] Metadata directory not provided
I20260928 06:48:27.732409 23384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:48:27.736344 23384 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 06:48:27.738648 23419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:27.739986 23384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:48:27.740127 23384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "054e90aa690748eca63669817f350244"
format_stamp: "Formatted at 2026-09-28 06:48:06 on dist-test-slave-twl6"
server_key: "d07d5b40d0508bad8a1bc4b81119fc04"
server_key_iv: "1e4af7bbd3177d44050d936216ca6fa3"
server_key_version: "encryptionkey@0"
I20260928 06:48:27.740681 23384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:48:27.754138 23384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:48:27.757330 23384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:48:27.757540 23384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:48:27.758075 23384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:48:27.759099 23426 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:48:27.763912 23384 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:48:27.763983 23384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260928 06:48:27.764051 23384 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:48:27.765605 23384 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:48:27.765667 23384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 06:48:27.765928 23426 tablet_bootstrap.cc:492] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244: Bootstrap starting.
I20260928 06:48:27.775879 23384 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:40317
I20260928 06:48:27.775949 23533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:40317 every 8 connection(s)
I20260928 06:48:27.776738 23384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:48:27.784032 23426 log.cc:824] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244: Log is configured to *not* fsync() on all Append() calls
I20260928 06:48:27.784922 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 23384
I20260928 06:48:27.785380 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:45701
--local_ip_for_outbound_sockets=127.7.48.2
--tserver_master_addrs=127.7.48.62:33141
--webserver_port=42515
--webserver_interface=127.7.48.2
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:48:27 dist-test-slave-twl6 krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578107, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:48:27.791652 23426 tablet_bootstrap.cc:492] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:48:27.792071 23426 tablet_bootstrap.cc:492] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244: Bootstrap complete.
I20260928 06:48:27.793293 23426 ts_tablet_manager.cc:1397] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244: Time spent bootstrapping tablet: real 0.028s user 0.018s sys 0.008s
I20260928 06:48:27.794067 23536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:27.778701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.1:37799)
0928 06:48:27.779278 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:27.779353 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 06:48:27.780419 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:48:27.781460 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:48:27.781468 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:48:27.781961 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:48:27.782574 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:27.782592 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:27.783720 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:27.783723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:27.784936 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:27.784948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:27.785912 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:27.785915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:27.786098 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:27.786728 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:48:27.786746 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:48:27.788195 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:48:27.790724 (+ 2529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:48:27.790729 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:48:27.790739 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:48:27.791035 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:48:27.791431 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:48:27.791437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:48:27.791441 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:48:27.791603 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:48:27.792361 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:48:27.792369 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:48:27.792655 (+ 286us) client_negotiation.cc:769] Sending connection context
0928 06:48:27.792830 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 06:48:27.793145 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":126,"threads_started":1}
I20260928 06:48:27.794275 23537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:27.778777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:37799 (local address 127.7.48.62:33141)
0928 06:48:27.779632 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0928 06:48:27.779646 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:48:27.779681 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:48:27.780809 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:48:27.780819 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:48:27.781107 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:48:27.781345 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:48:27.782722 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:27.783581 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:27.785087 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:27.785769 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:27.785944 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:27.788364 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:48:27.788368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:48:27.788378 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:48:27.788400 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:48:27.790592 (+ 2192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:27.791165 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:27.791172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:27.791180 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:27.791326 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:27.791712 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:27.791718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:27.791722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:27.792227 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:48:27.792364 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 06:48:27.792995 (+ 631us) server_negotiation.cc:299] Negotiation successful
0928 06:48:27.793315 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":132,"threads_started":1}
I20260928 06:48:27.796128 23426 raft_consensus.cc:348] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } }
I20260928 06:48:27.796660 23426 raft_consensus.cc:729] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 054e90aa690748eca63669817f350244, State: Initialized, Role: FOLLOWER
I20260928 06:48:27.796703 23534 heartbeater.cc:347] Connected to a master server at 127.7.48.62:33141
I20260928 06:48:27.796960 23534 heartbeater.cc:464] Registering TS with master...
I20260928 06:48:27.797174 23426 consensus_queue.cc:260] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } }
I20260928 06:48:27.797559 23534 heartbeater.cc:511] Master 127.7.48.62:33141 requested a full tablet report, sending...
I20260928 06:48:27.797973 23426 ts_tablet_manager.cc:1428] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 06:48:27.800046 23346 ts_manager.cc:194] Registered new tserver with Master: 054e90aa690748eca63669817f350244 (127.7.48.1:40317)
I20260928 06:48:27.803144 23346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.1@KRBTEST.COM'} at 127.7.48.1:37799
I20260928 06:48:27.805452 23534 heartbeater.cc:503] Master 127.7.48.62:33141 was elected leader, sending a full tablet report...
Sep 28 06:48:27 dist-test-slave-twl6 krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578107, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for kudu/127.7.48.1@KRBTEST.COM
I20260928 06:48:27.813772 23542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:27.802706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41752 (local address 127.7.48.1:40317)
0928 06:48:27.803355 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0928 06:48:27.803368 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:48:27.803392 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:48:27.803461 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:48:27.803469 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:48:27.803652 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:48:27.803922 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:48:27.805645 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:27.806596 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:27.807420 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:27.807909 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:27.808015 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:27.810277 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:48:27.810287 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:48:27.810299 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:48:27.810324 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:48:27.811965 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:27.812456 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:27.812460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:27.812461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:27.812563 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:27.812826 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:27.812828 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:27.812829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:27.813044 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:48:27.813147 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 06:48:27.813510 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 06:48:27.813652 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":103,"threads_started":1}
I20260928 06:48:27.813802 23541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:27.802614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:40317 (local address 127.0.0.1:41752)
0928 06:48:27.803159 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:27.803200 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:48:27.803311 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:48:27.803933 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:48:27.803938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:48:27.804258 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:48:27.804348 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:27.804354 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:27.806718 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:27.806721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:27.807317 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:27.807326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:27.808006 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:27.808019 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:27.808112 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:27.808666 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:48:27.808678 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:48:27.810143 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:48:27.812068 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:48:27.812077 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:48:27.812082 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:48:27.812356 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:48:27.812646 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:48:27.812649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:48:27.812651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:48:27.812741 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:48:27.813154 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:48:27.813159 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:48:27.813289 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 06:48:27.813435 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 06:48:27.813641 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":97,"threads_started":1}
I20260928 06:48:27.814754 23468 tablet_service.cc:1514] Processing DeleteTablet for tablet fabfc261ae2f4de29eb108c47f5211ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:48:07 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:41752
I20260928 06:48:27.815577 23543 tablet_replica.cc:330] stopping tablet replica
I20260928 06:48:27.815904 23543 raft_consensus.cc:2229] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:48:27.816188 23543 raft_consensus.cc:2258] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:48:27.817200 23543 ts_tablet_manager.cc:1905] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:48:27.821496 23543 ts_tablet_manager.cc:1918] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:48:27.821593 23543 log.cc:1196] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/wal/wals/fabfc261ae2f4de29eb108c47f5211ac
I20260928 06:48:27.821897 23543 ts_tablet_manager.cc:1939] T fabfc261ae2f4de29eb108c47f5211ac P 054e90aa690748eca63669817f350244: Deleting consensus metadata
I20260928 06:48:27.823925 23330 catalog_manager.cc:4841] TS 054e90aa690748eca63669817f350244 (127.7.48.1:40317): tablet fabfc261ae2f4de29eb108c47f5211ac (table test-table [id=d421815d84334360af103c3eb97c2339]) successfully deleted
W20260928 06:48:27.899003 23539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:48:27.899251 23539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:48:27.899292 23539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:48:27.903620 23539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:48:27.903680 23539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:48:27.903748 23539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:48:27.908710 23539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:45701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=42515
--tserver_master_addrs=127.7.48.62:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:48:27.909652 23539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:48:27.910571 23539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:48:27.919454 23549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:48:27.919480 23548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:48:27.919551 23539 server_base.cc:1034] running on GCE node
W20260928 06:48:27.919469 23551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:48:27.920171 23539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:48:27.920732 23539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:48:27.921878 23539 hybrid_clock.cc:648] HybridClock initialized: now 1790578107921852 us; error 39 us; skew 500 ppm
Sep 28 06:48:27 dist-test-slave-twl6 krb5kdc[22765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578107, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:48:27.924371 23539 init.cc:377] Logged in from keytab as kudu/127.7.48.2@KRBTEST.COM (short username kudu)
I20260928 06:48:27.925243 23539 webserver.cc:460] Webserver started at http://127.7.48.2:42515/ using document root <none> and password file <none>
I20260928 06:48:27.925772 23539 fs_manager.cc:359] Metadata directory not provided
I20260928 06:48:27.925864 23539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:48:27.929705 23539 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:48:27.931505 23558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:27.932622 23539 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:48:27.932740 23539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "20932349d3e147af931d4d553f6005db"
format_stamp: "Formatted at 2026-09-28 06:48:06 on dist-test-slave-twl6"
server_key: "e8fbbc455c54bf8904e3935099b89268"
server_key_iv: "f49044de2e589fef6ae122a25c1dd437"
server_key_version: "encryptionkey@0"
I20260928 06:48:27.933267 23539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:48:27.956941 23539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:48:27.959669 23539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:48:27.959895 23539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:48:27.960476 23539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:48:27.961234 23539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:48:27.961304 23539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:27.961366 23539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:48:27.961395 23539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:27.969834 23539 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:45701
I20260928 06:48:27.969859 23671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:45701 every 8 connection(s)
I20260928 06:48:27.970752 23539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:48:27.973351 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 23539
I20260928 06:48:27.973788 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:43811
--local_ip_for_outbound_sockets=127.7.48.3
--tserver_master_addrs=127.7.48.62:33141
--webserver_port=35789
--webserver_interface=127.7.48.3
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:48:27 dist-test-slave-twl6 krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578107, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:48:27.984160 23537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:27.972353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:33807 (local address 127.7.48.62:33141)
0928 06:48:27.972470 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 06:48:27.972474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:48:27.973197 (+ 723us) server_negotiation.cc:407] Connection header received
0928 06:48:27.974562 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:48:27.974565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:48:27.974631 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:48:27.974730 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:48:27.975805 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:27.976337 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:27.977446 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:27.977660 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:27.977759 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:27.980493 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:48:27.980496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:48:27.980497 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:48:27.980522 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:48:27.981835 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:27.982513 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:27.982516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:27.982517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:27.982595 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:27.982987 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:27.982990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:27.982991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:27.983163 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:48:27.983232 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:48:27.983853 (+ 621us) server_negotiation.cc:299] Negotiation successful
0928 06:48:27.984046 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:48:27.985098 23674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:27.972669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.2:33807)
0928 06:48:27.973064 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:27.973102 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:48:27.974365 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:48:27.974893 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:48:27.974899 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:48:27.975222 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:48:27.975674 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:27.975686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:27.976462 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:27.976465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:27.977329 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:27.977337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:27.977795 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:27.977799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:27.977949 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:27.978468 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:48:27.978483 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:48:27.980325 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:48:27.981970 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:48:27.981979 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:48:27.981995 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:48:27.982404 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:48:27.982686 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:48:27.982692 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:48:27.982697 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:48:27.982892 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:48:27.983266 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:48:27.983285 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:48:27.983608 (+ 323us) client_negotiation.cc:769] Sending connection context
0928 06:48:27.983815 (+ 207us) client_negotiation.cc:240] Negotiation successful
0928 06:48:27.984198 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":108,"threads_started":1}
I20260928 06:48:27.986171 23672 heartbeater.cc:347] Connected to a master server at 127.7.48.62:33141
I20260928 06:48:27.986409 23672 heartbeater.cc:464] Registering TS with master...
I20260928 06:48:27.986842 23672 heartbeater.cc:511] Master 127.7.48.62:33141 requested a full tablet report, sending...
I20260928 06:48:27.987856 23346 ts_manager.cc:194] Registered new tserver with Master: 20932349d3e147af931d4d553f6005db (127.7.48.2:45701)
I20260928 06:48:27.988421 23346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.2@KRBTEST.COM'} at 127.7.48.2:33807
W20260928 06:48:28.079128 23675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:48:28.079361 23675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:48:28.079439 23675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:48:28.083034 23675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:48:28.083081 23675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:48:28.083179 23675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:48:28.087848 23675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:43811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=35789
--tserver_master_addrs=127.7.48.62:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:48:28.088795 23675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:48:28.089749 23675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:48:28.098240 23681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:48:28.098263 23680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:48:28.098317 23675 server_base.cc:1034] running on GCE node
W20260928 06:48:28.098276 23683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:48:28.098855 23675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:48:28.099400 23675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:48:28.100576 23675 hybrid_clock.cc:648] HybridClock initialized: now 1790578108100546 us; error 42 us; skew 500 ppm
Sep 28 06:48:28 dist-test-slave-twl6 krb5kdc[22765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578108, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:48:28.103071 23675 init.cc:377] Logged in from keytab as kudu/127.7.48.3@KRBTEST.COM (short username kudu)
I20260928 06:48:28.103791 23675 webserver.cc:460] Webserver started at http://127.7.48.3:35789/ using document root <none> and password file <none>
I20260928 06:48:28.104310 23675 fs_manager.cc:359] Metadata directory not provided
I20260928 06:48:28.104422 23675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:48:28.107738 23675 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:48:28.109671 23690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:28.110778 23675 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:48:28.110913 23675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "b5c2c05c3b944b8b9d1f8889988ad296"
format_stamp: "Formatted at 2026-09-28 06:48:06 on dist-test-slave-twl6"
server_key: "6ec4d015935492f95f28443d53b0381e"
server_key_iv: "9b87daf790ba59bde6c5199748b94b01"
server_key_version: "encryptionkey@0"
I20260928 06:48:28.111402 23675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:48:28.127351 23675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:48:28.130390 23675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:48:28.130564 23675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:48:28.131084 23675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:48:28.131841 23675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:48:28.131958 23675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:28.132037 23675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:48:28.132077 23675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:28.140806 23675 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:43811
I20260928 06:48:28.140861 23806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:43811 every 8 connection(s)
I20260928 06:48:28.141685 23675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:48:28.149389 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 23675
Sep 28 06:48:28 dist-test-slave-twl6 krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578108, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:48:28.154850 23537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:28.143279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:42523 (local address 127.7.48.62:33141)
0928 06:48:28.143391 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 06:48:28.143394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:48:28.144208 (+ 814us) server_negotiation.cc:407] Connection header received
0928 06:48:28.145388 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:48:28.145391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:48:28.145445 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:48:28.145530 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:48:28.146566 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.147045 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.148147 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.148372 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.148475 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:28.151178 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:48:28.151184 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:48:28.151187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:48:28.151216 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:48:28.152784 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:28.153200 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:28.153203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:28.153205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:28.153298 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:28.153665 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:28.153668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:28.153669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:28.153892 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:48:28.153991 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 06:48:28.154518 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 06:48:28.154638 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:48:28.155673 23809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:28.143596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.3:42523)
0928 06:48:28.144076 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:28.144119 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:48:28.145205 (+ 1086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:48:28.145681 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:48:28.145705 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:48:28.146008 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:48:28.146430 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:28.146442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.147159 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.147162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:28.148010 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:28.148019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.148512 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.148517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:28.148689 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:28.149215 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:48:28.149239 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:48:28.151029 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:48:28.152879 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:48:28.152883 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:48:28.152892 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:48:28.153116 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:48:28.153378 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:48:28.153381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:48:28.153382 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:48:28.153572 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:48:28.153993 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:48:28.153998 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:48:28.154248 (+ 250us) client_negotiation.cc:769] Sending connection context
0928 06:48:28.154449 (+ 201us) client_negotiation.cc:240] Negotiation successful
0928 06:48:28.154688 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":123,"threads_started":1}
I20260928 06:48:28.156906 23807 heartbeater.cc:347] Connected to a master server at 127.7.48.62:33141
I20260928 06:48:28.157145 23807 heartbeater.cc:464] Registering TS with master...
I20260928 06:48:28.157693 23807 heartbeater.cc:511] Master 127.7.48.62:33141 requested a full tablet report, sending...
I20260928 06:48:28.158705 23810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:28.150025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49070 (local address 127.7.48.62:33141)
0928 06:48:28.150221 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 06:48:28.150224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:48:28.150494 (+ 270us) server_negotiation.cc:407] Connection header received
0928 06:48:28.150692 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:48:28.150695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:48:28.150747 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:48:28.150845 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:48:28.151747 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.152399 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.153380 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.153726 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.153796 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:28.155339 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:48:28.155342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:48:28.155344 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:48:28.155367 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:48:28.157101 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:28.157520 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:28.157522 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:28.157523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:28.157587 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:28.157870 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:28.157873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:28.157875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:28.158054 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:48:28.158257 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 06:48:28.158382 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 06:48:28.158506 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"spinlock_wait_cycles":19200,"thread_start_us":76,"threads_started":1}
I20260928 06:48:28.158793 23346 ts_manager.cc:194] Registered new tserver with Master: b5c2c05c3b944b8b9d1f8889988ad296 (127.7.48.3:43811)
I20260928 06:48:28.159412 23346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.3@KRBTEST.COM'} at 127.7.48.3:42523
I20260928 06:48:28.160413 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:48:28.164389 23810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:28.160960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49086 (local address 127.7.48.62:33141)
0928 06:48:28.161064 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 06:48:28.161068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:48:28.161598 (+ 530us) server_negotiation.cc:407] Connection header received
0928 06:48:28.161725 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:48:28.161728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:48:28.161812 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:48:28.161904 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:48:28.162110 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.162551 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.163308 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.163574 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.163678 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:28.163957 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:48:28.164062 (+ 105us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 06:48:28.164288 (+ 226us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49086: token expired
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 06:48:28.164464 23810 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49086: token expired
W20260928 06:48:28.164729 23272 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.7.48.62:33141)
I20260928 06:48:28.171187 23810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:28.165040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49102 (local address 127.7.48.62:33141)
0928 06:48:28.165141 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 06:48:28.165144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:48:28.165210 (+ 66us) server_negotiation.cc:407] Connection header received
0928 06:48:28.165311 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:48:28.165314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:48:28.165352 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:48:28.165426 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:48:28.165970 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.166376 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.167077 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.167262 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.167353 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:28.168517 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:48:28.168521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:48:28.168523 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:48:28.168549 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:48:28.169732 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:28.170133 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:28.170137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:28.170139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:28.170213 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:28.170481 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:28.170484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:28.170485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:28.170632 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:48:28.170749 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 06:48:28.170951 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 06:48:28.171054 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:48:28.173178 23345 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:48:28.176245 23345 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:48:28.183465 23468 tablet_service.cc:1467] Processing CreateTablet for tablet eafa2e0eb1034cada1004f2e35d45a72 (DEFAULT_TABLE table=test-table [id=8a0354df187b4eefaef34669861c8dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:48:28.184131 23468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eafa2e0eb1034cada1004f2e35d45a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:48:28.186551 23426 tablet_bootstrap.cc:492] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244: Bootstrap starting.
I20260928 06:48:28.187420 23426 tablet_bootstrap.cc:654] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244: Neither blocks nor log segments found. Creating new log.
I20260928 06:48:28.187443 23541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:28.182263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:45701 (local address 127.0.0.1:51302)
0928 06:48:28.182418 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:28.182432 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:48:28.182531 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:48:28.183362 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:48:28.183365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:48:28.183382 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:48:28.183472 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:28.183478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.184704 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.184707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:28.185916 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:28.185926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.187001 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.187004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:28.187104 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:28.187167 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 06:48:28.187236 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:48:28.187317 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260928 06:48:28.187698 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:28.182443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:43811 (local address 127.0.0.1:41856)
0928 06:48:28.182678 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:28.182691 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:48:28.182775 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:48:28.183781 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:48:28.183783 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:48:28.183799 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:48:28.183926 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:28.183933 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.185034 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.185037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:28.186095 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:28.186105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.187325 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.187328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:28.187418 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:28.187438 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 06:48:28.187505 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:48:28.187582 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":88,"threads_started":1}
I20260928 06:48:28.188047 23817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:28.182451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51302 (local address 127.7.48.2:45701)
0928 06:48:28.182913 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0928 06:48:28.182922 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:48:28.182940 (+ 18us) server_negotiation.cc:407] Connection header received
0928 06:48:28.182998 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:48:28.183001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:48:28.183220 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:48:28.183373 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:48:28.183581 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.184581 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.186059 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.186873 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.186979 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:28.187138 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 06:48:28.187783 (+ 645us) server_negotiation.cc:299] Negotiation successful
0928 06:48:28.187914 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":130,"threads_started":1}
I20260928 06:48:28.188170 23818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:28.182853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41856 (local address 127.7.48.3:43811)
0928 06:48:28.183182 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0928 06:48:28.183194 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:48:28.183214 (+ 20us) server_negotiation.cc:407] Connection header received
0928 06:48:28.183280 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:48:28.183284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:48:28.183483 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:48:28.183618 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:48:28.184040 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.184880 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.186355 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.187031 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.187138 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:28.187269 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 06:48:28.187929 (+ 660us) server_negotiation.cc:299] Negotiation successful
0928 06:48:28.188024 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":161,"threads_started":1}
I20260928 06:48:28.188990 23426 tablet_bootstrap.cc:492] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244: No bootstrap required, opened a new log
I20260928 06:48:28.189088 23426 ts_tablet_manager.cc:1397] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:48:28.189508 23426 raft_consensus.cc:348] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } }
I20260928 06:48:28.189628 23426 raft_consensus.cc:374] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:48:28.189666 23426 raft_consensus.cc:729] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 054e90aa690748eca63669817f350244, State: Initialized, Role: FOLLOWER
I20260928 06:48:28.189790 23426 consensus_queue.cc:260] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } }
I20260928 06:48:28.190037 23426 ts_tablet_manager.cc:1428] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:48:28.191042 23739 tablet_service.cc:1467] Processing CreateTablet for tablet eafa2e0eb1034cada1004f2e35d45a72 (DEFAULT_TABLE table=test-table [id=8a0354df187b4eefaef34669861c8dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:48:28.192045 23739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eafa2e0eb1034cada1004f2e35d45a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:48:28.197883 23606 tablet_service.cc:1467] Processing CreateTablet for tablet eafa2e0eb1034cada1004f2e35d45a72 (DEFAULT_TABLE table=test-table [id=8a0354df187b4eefaef34669861c8dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:48:28.198743 23820 tablet_bootstrap.cc:492] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296: Bootstrap starting.
I20260928 06:48:28.198959 23606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eafa2e0eb1034cada1004f2e35d45a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:48:28.200795 23820 tablet_bootstrap.cc:654] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296: Neither blocks nor log segments found. Creating new log.
I20260928 06:48:28.201856 23820 log.cc:824] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296: Log is configured to *not* fsync() on all Append() calls
I20260928 06:48:28.203456 23820 tablet_bootstrap.cc:492] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296: No bootstrap required, opened a new log
I20260928 06:48:28.203649 23820 ts_tablet_manager.cc:1397] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 06:48:28.204934 23822 tablet_bootstrap.cc:492] T eafa2e0eb1034cada1004f2e35d45a72 P 20932349d3e147af931d4d553f6005db: Bootstrap starting.
I20260928 06:48:28.206624 23820 raft_consensus.cc:348] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } }
I20260928 06:48:28.206905 23820 raft_consensus.cc:374] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:48:28.206964 23820 raft_consensus.cc:729] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c2c05c3b944b8b9d1f8889988ad296, State: Initialized, Role: FOLLOWER
I20260928 06:48:28.207171 23822 tablet_bootstrap.cc:654] T eafa2e0eb1034cada1004f2e35d45a72 P 20932349d3e147af931d4d553f6005db: Neither blocks nor log segments found. Creating new log.
I20260928 06:48:28.207368 23820 consensus_queue.cc:260] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } }
I20260928 06:48:28.207934 23822 log.cc:824] T eafa2e0eb1034cada1004f2e35d45a72 P 20932349d3e147af931d4d553f6005db: Log is configured to *not* fsync() on all Append() calls
I20260928 06:48:28.208001 23807 heartbeater.cc:503] Master 127.7.48.62:33141 was elected leader, sending a full tablet report...
I20260928 06:48:28.208344 23820 ts_tablet_manager.cc:1428] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 06:48:28.209934 23822 tablet_bootstrap.cc:492] T eafa2e0eb1034cada1004f2e35d45a72 P 20932349d3e147af931d4d553f6005db: No bootstrap required, opened a new log
I20260928 06:48:28.210107 23822 ts_tablet_manager.cc:1397] T eafa2e0eb1034cada1004f2e35d45a72 P 20932349d3e147af931d4d553f6005db: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 06:48:28.212874 23822 raft_consensus.cc:348] T eafa2e0eb1034cada1004f2e35d45a72 P 20932349d3e147af931d4d553f6005db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } }
I20260928 06:48:28.213163 23822 raft_consensus.cc:374] T eafa2e0eb1034cada1004f2e35d45a72 P 20932349d3e147af931d4d553f6005db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:48:28.213240 23822 raft_consensus.cc:729] T eafa2e0eb1034cada1004f2e35d45a72 P 20932349d3e147af931d4d553f6005db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20932349d3e147af931d4d553f6005db, State: Initialized, Role: FOLLOWER
I20260928 06:48:28.213676 23822 consensus_queue.cc:260] T eafa2e0eb1034cada1004f2e35d45a72 P 20932349d3e147af931d4d553f6005db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } }
I20260928 06:48:28.214250 23672 heartbeater.cc:503] Master 127.7.48.62:33141 was elected leader, sending a full tablet report...
I20260928 06:48:28.214526 23822 ts_tablet_manager.cc:1428] T eafa2e0eb1034cada1004f2e35d45a72 P 20932349d3e147af931d4d553f6005db: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260928 06:48:28.221598 23673 tablet.cc:2369] T eafa2e0eb1034cada1004f2e35d45a72 P 20932349d3e147af931d4d553f6005db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:48:28.256932 23823 raft_consensus.cc:482] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:48:28.257138 23823 raft_consensus.cc:504] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } }
I20260928 06:48:28.258281 23823 leader_election.cc:288] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20932349d3e147af931d4d553f6005db (127.7.48.2:45701), 054e90aa690748eca63669817f350244 (127.7.48.1:40317)
I20260928 06:48:28.261943 23809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:28.258514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:45701 (local address 127.7.48.3:59699)
0928 06:48:28.258658 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:28.258687 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 06:48:28.258792 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:48:28.259135 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:48:28.259138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:48:28.259155 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:48:28.259289 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:28.259296 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.260219 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.260225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:28.260928 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:28.260936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.261552 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.261556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:28.261654 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:28.261679 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 06:48:28.261755 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:48:28.261822 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":1}
I20260928 06:48:28.261943 23817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:28.258639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:59699 (local address 127.7.48.2:45701)
0928 06:48:28.258768 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 06:48:28.258773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:48:28.258787 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:48:28.258937 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:48:28.258942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:48:28.259004 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:48:28.259086 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:48:28.259386 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.260129 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.261039 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.261444 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.261552 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:28.261621 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:48:28.261769 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 06:48:28.261832 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 06:48:28.262496 23826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:28.258826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:40317 (local address 127.7.48.3:37875)
0928 06:48:28.259174 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:28.259185 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:48:28.259329 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:48:28.259677 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:48:28.259680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:48:28.259698 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:48:28.259821 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:28.259827 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.260606 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.260610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:28.261365 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:28.261374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.261986 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.262110 (+ 124us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:28.262200 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:28.262220 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 06:48:28.262310 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 06:48:28.262374 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":128,"threads_started":1}
I20260928 06:48:28.262583 23626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eafa2e0eb1034cada1004f2e35d45a72" candidate_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20932349d3e147af931d4d553f6005db" is_pre_election: true
I20260928 06:48:28.262874 23626 raft_consensus.cc:2454] T eafa2e0eb1034cada1004f2e35d45a72 P 20932349d3e147af931d4d553f6005db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5c2c05c3b944b8b9d1f8889988ad296 in term 0.
I20260928 06:48:28.263360 23542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:28.258933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:37875 (local address 127.7.48.1:40317)
0928 06:48:28.259074 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 06:48:28.259078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:48:28.259298 (+ 220us) server_negotiation.cc:407] Connection header received
0928 06:48:28.259475 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:48:28.259478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:48:28.259547 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:48:28.259649 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:48:28.259953 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.260503 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.261471 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.261855 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.261960 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:28.262141 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 06:48:28.263136 (+ 995us) server_negotiation.cc:299] Negotiation successful
0928 06:48:28.263224 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 06:48:28.263329 23691 leader_election.cc:302] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20932349d3e147af931d4d553f6005db, b5c2c05c3b944b8b9d1f8889988ad296; no voters:
I20260928 06:48:28.263676 23823 raft_consensus.cc:2781] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:48:28.263767 23823 raft_consensus.cc:482] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:48:28.263820 23823 raft_consensus.cc:3037] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:48:28.263833 23488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eafa2e0eb1034cada1004f2e35d45a72" candidate_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "054e90aa690748eca63669817f350244" is_pre_election: true
I20260928 06:48:28.264130 23488 raft_consensus.cc:2454] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5c2c05c3b944b8b9d1f8889988ad296 in term 0.
I20260928 06:48:28.265014 23823 raft_consensus.cc:504] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } }
I20260928 06:48:28.265379 23823 leader_election.cc:288] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296 [CANDIDATE]: Term 1 election: Requested vote from peers 20932349d3e147af931d4d553f6005db (127.7.48.2:45701), 054e90aa690748eca63669817f350244 (127.7.48.1:40317)
I20260928 06:48:28.265640 23626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eafa2e0eb1034cada1004f2e35d45a72" candidate_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20932349d3e147af931d4d553f6005db"
I20260928 06:48:28.265769 23626 raft_consensus.cc:3037] T eafa2e0eb1034cada1004f2e35d45a72 P 20932349d3e147af931d4d553f6005db [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:48:28.265787 23488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eafa2e0eb1034cada1004f2e35d45a72" candidate_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "054e90aa690748eca63669817f350244"
I20260928 06:48:28.265928 23488 raft_consensus.cc:3037] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:48:28.266741 23626 raft_consensus.cc:2454] T eafa2e0eb1034cada1004f2e35d45a72 P 20932349d3e147af931d4d553f6005db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5c2c05c3b944b8b9d1f8889988ad296 in term 1.
I20260928 06:48:28.267014 23691 leader_election.cc:302] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20932349d3e147af931d4d553f6005db, b5c2c05c3b944b8b9d1f8889988ad296; no voters:
I20260928 06:48:28.267021 23488 raft_consensus.cc:2454] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5c2c05c3b944b8b9d1f8889988ad296 in term 1.
I20260928 06:48:28.267159 23823 raft_consensus.cc:2781] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:48:28.267390 23823 raft_consensus.cc:686] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296 [term 1 LEADER]: Becoming Leader. State: Replica: b5c2c05c3b944b8b9d1f8889988ad296, State: Running, Role: LEADER
I20260928 06:48:28.267692 23823 consensus_queue.cc:237] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } }
I20260928 06:48:28.270737 23346 catalog_manager.cc:5495] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5c2c05c3b944b8b9d1f8889988ad296 (127.7.48.3). New cstate: current_term: 1 leader_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } health_report { overall_health: UNKNOWN } } }
W20260928 06:48:28.277813 23535 tablet.cc:2369] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:48:28.287556 23818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:28.284357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41860 (local address 127.7.48.3:43811)
0928 06:48:28.284487 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 06:48:28.284492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:48:28.284506 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:48:28.284673 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:48:28.284675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:48:28.284728 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:48:28.284828 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:48:28.285041 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.285518 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.286306 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:28.286498 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:28.286595 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:28.286854 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:48:28.287221 (+ 367us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:48:28.287340 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 06:48:28.287377 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 06:48:28.287427 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:48:28.293089 23488 raft_consensus.cc:1261] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Refusing update from remote peer b5c2c05c3b944b8b9d1f8889988ad296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:48:28.293099 23626 raft_consensus.cc:1261] T eafa2e0eb1034cada1004f2e35d45a72 P 20932349d3e147af931d4d553f6005db [term 1 FOLLOWER]: Refusing update from remote peer b5c2c05c3b944b8b9d1f8889988ad296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:48:28.293807 23827 consensus_queue.cc:1035] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:48:28.293996 23823 consensus_queue.cc:1035] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:48:28.299819 23830 mvcc.cc:205] Tried to move back new op lower bound from 7334207931561861120 to 7334207931466878976. Current Snapshot: MvccSnapshot[applied={T|T < 7334207931561861120}]
I20260928 06:48:28.300328 23832 mvcc.cc:205] Tried to move back new op lower bound from 7334207931561861120 to 7334207931466878976. Current Snapshot: MvccSnapshot[applied={T|T < 7334207931561861120}]
I20260928 06:48:28.303197 23831 mvcc.cc:205] Tried to move back new op lower bound from 7334207931561861120 to 7334207931466878976. Current Snapshot: MvccSnapshot[applied={T|T < 7334207931561861120}]
I20260928 06:48:28.309345 23345 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:48:28.309482 23345 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:48:28.311553 23345 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f: Sending DeleteTablet for 3 replicas of tablet eafa2e0eb1034cada1004f2e35d45a72
I20260928 06:48:28.312204 23606 tablet_service.cc:1514] Processing DeleteTablet for tablet eafa2e0eb1034cada1004f2e35d45a72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:48:28 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:51302
I20260928 06:48:28.312276 23739 tablet_service.cc:1514] Processing DeleteTablet for tablet eafa2e0eb1034cada1004f2e35d45a72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:48:28 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:41856
I20260928 06:48:28.312354 23468 tablet_service.cc:1514] Processing DeleteTablet for tablet eafa2e0eb1034cada1004f2e35d45a72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:48:28 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:41752
I20260928 06:48:28.312516 23543 tablet_replica.cc:330] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244: stopping tablet replica
I20260928 06:48:28.312635 23543 raft_consensus.cc:2229] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:48:28.312701 23841 tablet_replica.cc:330] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296: stopping tablet replica
I20260928 06:48:28.312758 23543 raft_consensus.cc:2258] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:48:28.312969 23841 raft_consensus.cc:2229] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:48:28.312991 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 23384
I20260928 06:48:28.313323 23841 raft_consensus.cc:2258] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:48:28.313709 23840 tablet_replica.cc:330] T eafa2e0eb1034cada1004f2e35d45a72 P 20932349d3e147af931d4d553f6005db: stopping tablet replica
I20260928 06:48:28.313970 23840 raft_consensus.cc:2229] T eafa2e0eb1034cada1004f2e35d45a72 P 20932349d3e147af931d4d553f6005db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:48:28.314245 23840 raft_consensus.cc:2258] T eafa2e0eb1034cada1004f2e35d45a72 P 20932349d3e147af931d4d553f6005db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:48:28.314765 23841 ts_tablet_manager.cc:1905] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:48:28.316565 23840 ts_tablet_manager.cc:1905] T eafa2e0eb1034cada1004f2e35d45a72 P 20932349d3e147af931d4d553f6005db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:48:28.317716 23841 ts_tablet_manager.cc:1918] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:48:28.317808 23841 log.cc:1196] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/eafa2e0eb1034cada1004f2e35d45a72
I20260928 06:48:28.318156 23841 ts_tablet_manager.cc:1939] T eafa2e0eb1034cada1004f2e35d45a72 P b5c2c05c3b944b8b9d1f8889988ad296: Deleting consensus metadata
I20260928 06:48:28.319250 23840 ts_tablet_manager.cc:1918] T eafa2e0eb1034cada1004f2e35d45a72 P 20932349d3e147af931d4d553f6005db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:48:28.319355 23840 log.cc:1196] T eafa2e0eb1034cada1004f2e35d45a72 P 20932349d3e147af931d4d553f6005db: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/eafa2e0eb1034cada1004f2e35d45a72
I20260928 06:48:28.319541 23331 catalog_manager.cc:4841] TS b5c2c05c3b944b8b9d1f8889988ad296 (127.7.48.3:43811): tablet eafa2e0eb1034cada1004f2e35d45a72 (table test-table [id=8a0354df187b4eefaef34669861c8dfc]) successfully deleted
I20260928 06:48:28.319682 23840 ts_tablet_manager.cc:1939] T eafa2e0eb1034cada1004f2e35d45a72 P 20932349d3e147af931d4d553f6005db: Deleting consensus metadata
I20260928 06:48:28.320592 23330 catalog_manager.cc:4841] TS 20932349d3e147af931d4d553f6005db (127.7.48.2:45701): tablet eafa2e0eb1034cada1004f2e35d45a72 (table test-table [id=8a0354df187b4eefaef34669861c8dfc]) successfully deleted
W20260928 06:48:28.321727 23330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.48.1:40317 (error 108)
I20260928 06:48:28.321841 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 23539
I20260928 06:48:28.324074 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:28.323776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:40317 (local address 127.0.0.1:41758)
0928 06:48:28.323913 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:28.323983 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.1:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 06:48:28.324210 23330 catalog_manager.cc:4584] TS 054e90aa690748eca63669817f350244 (127.7.48.1:40317): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eafa2e0eb1034cada1004f2e35d45a72: Network error: Client connection negotiation failed: client connection to 127.7.48.1:40317: connect: Connection refused (error 111)
I20260928 06:48:28.328823 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 23675
I20260928 06:48:28.334216 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 23313
I20260928 06:48:48.340852 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:33141
--webserver_interface=127.7.48.62
--webserver_port=41089
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:33141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:48:48.450465 23843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:48:48.450731 23843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:48:48.450771 23843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:48:48.454607 23843 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 06:48:48.454665 23843 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 06:48:48.454691 23843 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 06:48:48.454704 23843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:48:48.454737 23843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:48:48.454754 23843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:48:48.454769 23843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:48:48.454784 23843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:48:48.460001 23843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:33141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:33141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=41089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:48:48.461123 23843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:48:48.462168 23843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:48:48.470431 23851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:48:48.470439 23849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:48:48.470518 23843 server_base.cc:1034] running on GCE node
W20260928 06:48:48.470443 23848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:48:48.471285 23843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:48:48.472425 23843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:48:48.473611 23843 hybrid_clock.cc:648] HybridClock initialized: now 1790578128473579 us; error 47 us; skew 500 ppm
Sep 28 06:48:48 dist-test-slave-twl6 krb5kdc[22765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578128, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:48:48.480057 23843 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:48:48.481062 23843 webserver.cc:460] Webserver started at http://127.7.48.62:41089/ using document root <none> and password file <none>
I20260928 06:48:48.481673 23843 fs_manager.cc:359] Metadata directory not provided
I20260928 06:48:48.481766 23843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:48:48.485699 23843 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:48:48.487813 23858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:48.489092 23843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:48:48.489256 23843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "52938bc4827343ae836c286f671af74f"
format_stamp: "Formatted at 2026-09-28 06:48:06 on dist-test-slave-twl6"
server_key: "4608f488d78d9b4f78aa11fbc9eec076"
server_key_iv: "2a3e05411162a03641ae8209db572de0"
server_key_version: "encryptionkey@0"
I20260928 06:48:48.489797 23843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:48:48.512159 23843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:48:48.515582 23843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:48:48.515792 23843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:48:48.523985 23843 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:33141
I20260928 06:48:48.524322 23910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:33141 every 8 connection(s)
I20260928 06:48:48.525488 23843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:48:48.527639 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 23843
I20260928 06:48:48.528182 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:40317
--local_ip_for_outbound_sockets=127.7.48.1
--tserver_master_addrs=127.7.48.62:33141
--webserver_port=34203
--webserver_interface=127.7.48.1
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:48:48.530540 23911 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:48:48.532298 23911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f: Bootstrap starting.
I20260928 06:48:48.549419 23911 log.cc:824] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:48:48.560473 23911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:48:48.560891 23911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f: Bootstrap complete.
I20260928 06:48:48.564891 23911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52938bc4827343ae836c286f671af74f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33141 } }
I20260928 06:48:48.565301 23911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52938bc4827343ae836c286f671af74f, State: Initialized, Role: FOLLOWER
I20260928 06:48:48.565796 23911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52938bc4827343ae836c286f671af74f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33141 } }
I20260928 06:48:48.565904 23911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:48:48.566016 23911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:48:48.566138 23911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [term 2 FOLLOWER]: Advancing to term 3
I20260928 06:48:48.568089 23911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52938bc4827343ae836c286f671af74f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33141 } }
I20260928 06:48:48.568413 23911 leader_election.cc:302] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52938bc4827343ae836c286f671af74f; no voters:
I20260928 06:48:48.568747 23911 leader_election.cc:288] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 06:48:48.568823 23915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [term 3 FOLLOWER]: Leader election won for term 3
I20260928 06:48:48.569010 23915 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [term 3 LEADER]: Becoming Leader. State: Replica: 52938bc4827343ae836c286f671af74f, State: Running, Role: LEADER
I20260928 06:48:48.569296 23915 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52938bc4827343ae836c286f671af74f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33141 } }
I20260928 06:48:48.569974 23911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:48:48.570192 23915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52938bc4827343ae836c286f671af74f. Latest consensus state: current_term: 3 leader_uuid: "52938bc4827343ae836c286f671af74f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52938bc4827343ae836c286f671af74f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33141 } } }
I20260928 06:48:48.570293 23915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [sys.catalog]: This master's current role is: LEADER
I20260928 06:48:48.570698 23919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:48:48.570832 23916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "52938bc4827343ae836c286f671af74f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52938bc4827343ae836c286f671af74f" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33141 } } }
I20260928 06:48:48.571014 23916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f [sys.catalog]: This master's current role is: LEADER
I20260928 06:48:48.576081 23919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:48:48.576745 23919 catalog_manager.cc:1259] Loaded cluster ID: d0ef55f99ef248aea06909b89a18ce86
I20260928 06:48:48.576807 23919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:48:48.579190 23919 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:48:48.579933 23919 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f: Loaded TSK: 0
I20260928 06:48:48.580011 23919 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 52938bc4827343ae836c286f671af74f: Loaded TSK: 1
F20260928 06:48:48.580739 23919 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb8945fb9ad google::LogMessage::Fail() at ??:0
@ 0x7fb8945fe88a google::LogMessage::SendToLog() at ??:0
@ 0x7fb8945fb4de google::LogMessage::Flush() at ??:0
@ 0x7fb8945fca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb8972bf18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb8972b7f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb89ef607bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb89ef3a99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb89ef3b178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fb89ef7cdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fb89ef9b82b std::function<>::operator()() at ??:0
@ 0x7fb89ef3ab4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fb89ef3bf3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb89ef38579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fb89ef7c640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb89684d3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb89684dc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fb89684f3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb89683e512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb897e826db start_thread at ??:0
@ 0x7fb8930f961f clone at ??:0
W20260928 06:48:48.650923 23913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:48:48.651141 23913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:48:48.651182 23913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:48:48.655491 23913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:48:48.655535 23913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:48:48.655591 23913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:48:48.661284 23913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:40317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=34203
--tserver_master_addrs=127.7.48.62:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:48:48.662292 23913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:48:48.663228 23913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:48:48.672994 23954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:48:48.672994 23957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:48:48.673213 23955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:48:48.673583 23913 server_base.cc:1034] running on GCE node
I20260928 06:48:48.674045 23913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:48:48.674651 23913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:48:48.675799 23913 hybrid_clock.cc:648] HybridClock initialized: now 1790578128675780 us; error 62 us; skew 500 ppm
Sep 28 06:48:48 dist-test-slave-twl6 krb5kdc[22765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578128, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:48:48.678629 23913 init.cc:377] Logged in from keytab as kudu/127.7.48.1@KRBTEST.COM (short username kudu)
I20260928 06:48:48.679430 23913 webserver.cc:460] Webserver started at http://127.7.48.1:34203/ using document root <none> and password file <none>
I20260928 06:48:48.679989 23913 fs_manager.cc:359] Metadata directory not provided
I20260928 06:48:48.680044 23913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:48:48.685230 23913 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 06:48:48.687338 23968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:48.688601 23913 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:48:48.688719 23913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "054e90aa690748eca63669817f350244"
format_stamp: "Formatted at 2026-09-28 06:48:06 on dist-test-slave-twl6"
server_key: "d07d5b40d0508bad8a1bc4b81119fc04"
server_key_iv: "1e4af7bbd3177d44050d936216ca6fa3"
server_key_version: "encryptionkey@0"
I20260928 06:48:48.689270 23913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:48:48.700721 23913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:48:48.703739 23913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:48:48.703979 23913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:48:48.704597 23913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:48:48.705634 23975 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:48:48.711004 23913 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:48:48.711099 23913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260928 06:48:48.711184 23913 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:48:48.712827 23913 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:48:48.712893 23913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260928 06:48:48.713183 23975 tablet_bootstrap.cc:492] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244: Bootstrap starting.
I20260928 06:48:48.723538 23913 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:40317
I20260928 06:48:48.723769 24082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:40317 every 8 connection(s)
I20260928 06:48:48.724774 23913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:48:48.734797 23975 log.cc:824] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244: Log is configured to *not* fsync() on all Append() calls
I20260928 06:48:48.734972 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 23913
I20260928 06:48:48.735466 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:45701
--local_ip_for_outbound_sockets=127.7.48.2
--tserver_master_addrs=127.7.48.62:33141
--webserver_port=42515
--webserver_interface=127.7.48.2
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:48:48.743543 23975 tablet_bootstrap.cc:492] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:48:48.743969 23975 tablet_bootstrap.cc:492] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244: Bootstrap complete.
I20260928 06:48:48.745383 23975 ts_tablet_manager.cc:1397] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244: Time spent bootstrapping tablet: real 0.032s user 0.021s sys 0.009s
I20260928 06:48:48.748747 23975 raft_consensus.cc:348] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } }
I20260928 06:48:48.749300 23975 raft_consensus.cc:729] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 054e90aa690748eca63669817f350244, State: Initialized, Role: FOLLOWER
I20260928 06:48:48.749809 23975 consensus_queue.cc:260] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } }
I20260928 06:48:48.750864 23975 ts_tablet_manager.cc:1428] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260928 06:48:48.817075 24085 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:48:48.726745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.1:37195)
0928 06:48:48.727248 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:48.727299 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:48:48.728140 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:48:48.815984 (+ 87844us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":163,"threads_started":1}
W20260928 06:48:48.817816 24083 heartbeater.cc:644] Failed to heartbeat to 127.7.48.62:33141 (0 consecutive failures): Network error: Failed to ping master at 127.7.48.62:33141: Client connection negotiation failed: client connection to 127.7.48.62:33141: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:48:48.818652 24085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:48.818301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.1:33845)
0928 06:48:48.818481 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:48.818562 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 06:48:48.819389 24085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:48.819155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.1:43279)
0928 06:48:48.819277 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:48.819331 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 06:48:48.819555 24083 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:48:48.848301 24087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:48:48.848549 24087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:48:48.848591 24087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:48:48.852484 24087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:48:48.852535 24087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:48:48.852593 24087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:48:48.857587 24087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:45701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=42515
--tserver_master_addrs=127.7.48.62:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:48:48.858579 24087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:48:48.859531 24087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:48:48.868820 24094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:48:48.868826 24093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:48:48.868830 24096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:48:48.869036 24087 server_base.cc:1034] running on GCE node
I20260928 06:48:48.869552 24087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:48:48.870198 24087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:48:48.871378 24087 hybrid_clock.cc:648] HybridClock initialized: now 1790578128871354 us; error 50 us; skew 500 ppm
Sep 28 06:48:48 dist-test-slave-twl6 krb5kdc[22765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578128, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:48:48.874199 24087 init.cc:377] Logged in from keytab as kudu/127.7.48.2@KRBTEST.COM (short username kudu)
I20260928 06:48:48.875047 24087 webserver.cc:460] Webserver started at http://127.7.48.2:42515/ using document root <none> and password file <none>
I20260928 06:48:48.875607 24087 fs_manager.cc:359] Metadata directory not provided
I20260928 06:48:48.875694 24087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:48:48.879467 24087 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:48:48.881628 24103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:48.882826 24087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:48:48.882956 24087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "20932349d3e147af931d4d553f6005db"
format_stamp: "Formatted at 2026-09-28 06:48:06 on dist-test-slave-twl6"
server_key: "e8fbbc455c54bf8904e3935099b89268"
server_key_iv: "f49044de2e589fef6ae122a25c1dd437"
server_key_version: "encryptionkey@0"
I20260928 06:48:48.883548 24087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:48:48.899180 24087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:48:48.902012 24087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:48:48.902216 24087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:48:48.902787 24087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:48:48.903589 24087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:48:48.903656 24087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:48.903716 24087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:48:48.903774 24087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:48.912861 24087 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:45701
I20260928 06:48:48.912896 24217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:45701 every 8 connection(s)
I20260928 06:48:48.913856 24087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:48:48.914098 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 24087
I20260928 06:48:48.914593 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:43811
--local_ip_for_outbound_sockets=127.7.48.3
--tserver_master_addrs=127.7.48.62:33141
--webserver_port=35789
--webserver_interface=127.7.48.3
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:48:48.918403 24221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:48.916219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.2:37021)
0928 06:48:48.916728 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:48.916878 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":115,"threads_started":1}
W20260928 06:48:48.918905 24218 heartbeater.cc:644] Failed to heartbeat to 127.7.48.62:33141 (0 consecutive failures): Network error: Failed to ping master at 127.7.48.62:33141: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
I20260928 06:48:48.919528 24221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:48.919321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.2:34683)
0928 06:48:48.919431 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:48.919475 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 06:48:48.920215 24221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:48.920038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.2:52445)
0928 06:48:48.920134 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:48.920173 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 06:48:48.920449 24218 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:48:49.023813 24220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:48:49.024145 24220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:48:49.024214 24220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:48:49.027949 24220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:48:49.028002 24220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:48:49.028098 24220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:48:49.033128 24220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:43811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=35789
--tserver_master_addrs=127.7.48.62:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:48:49.034166 24220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:48:49.035164 24220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:48:49.044625 24226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:48:49.044615 24229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:48:49.044615 24227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:48:49.044809 24220 server_base.cc:1034] running on GCE node
I20260928 06:48:49.045342 24220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:48:49.045921 24220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:48:49.047104 24220 hybrid_clock.cc:648] HybridClock initialized: now 1790578129047093 us; error 29 us; skew 500 ppm
Sep 28 06:48:49 dist-test-slave-twl6 krb5kdc[22765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578129, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:48:49.049849 24220 init.cc:377] Logged in from keytab as kudu/127.7.48.3@KRBTEST.COM (short username kudu)
I20260928 06:48:49.050747 24220 webserver.cc:460] Webserver started at http://127.7.48.3:35789/ using document root <none> and password file <none>
I20260928 06:48:49.051306 24220 fs_manager.cc:359] Metadata directory not provided
I20260928 06:48:49.051397 24220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:48:49.055125 24220 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:48:49.057168 24236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:49.058270 24220 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:48:49.058400 24220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "b5c2c05c3b944b8b9d1f8889988ad296"
format_stamp: "Formatted at 2026-09-28 06:48:06 on dist-test-slave-twl6"
server_key: "6ec4d015935492f95f28443d53b0381e"
server_key_iv: "9b87daf790ba59bde6c5199748b94b01"
server_key_version: "encryptionkey@0"
I20260928 06:48:49.058926 24220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:48:49.081336 24220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:48:49.084627 24220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:48:49.084834 24220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:48:49.085448 24220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:48:49.086228 24220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:48:49.086295 24220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:49.086361 24220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:48:49.086431 24220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:48:49.095355 24220 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:43811
I20260928 06:48:49.095659 24349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:43811 every 8 connection(s)
I20260928 06:48:49.096330 24220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:48:49.099503 24352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:49.098329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.3:50209)
0928 06:48:49.098752 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:49.098849 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":122,"threads_started":1}
W20260928 06:48:49.099967 24350 heartbeater.cc:644] Failed to heartbeat to 127.7.48.62:33141 (0 consecutive failures): Network error: Failed to ping master at 127.7.48.62:33141: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
I20260928 06:48:49.100533 24352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:49.100318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.3:57911)
0928 06:48:49.100426 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:49.100484 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 06:48:49.101082 24352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:49.100910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.3:53003)
0928 06:48:49.100999 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:49.101037 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 06:48:49.101248 24350 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:48:49.101883 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 24220
I20260928 06:48:49.820662 24354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:49.820219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.1:38245)
0928 06:48:49.820485 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:49.820545 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20260928 06:48:49.921345 24355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:49.920971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.2:36415)
0928 06:48:49.921197 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:49.921249 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":71,"threads_started":1}
I20260928 06:48:50.065542 24356 raft_consensus.cc:482] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:48:50.065725 24356 raft_consensus.cc:504] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } }
I20260928 06:48:50.066823 24356 leader_election.cc:288] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20932349d3e147af931d4d553f6005db (127.7.48.2:45701), b5c2c05c3b944b8b9d1f8889988ad296 (127.7.48.3:43811)
Sep 28 06:48:50 dist-test-slave-twl6 krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578128, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for kudu/127.7.48.2@KRBTEST.COM
Sep 28 06:48:50 dist-test-slave-twl6 krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578128, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for kudu/127.7.48.3@KRBTEST.COM
I20260928 06:48:50.082098 24357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:50.067006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:45701 (local address 127.7.48.1:41411)
0928 06:48:50.067673 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:50.067687 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:48:50.067802 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:48:50.069163 (+ 1361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:48:50.069168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:48:50.069706 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:48:50.070182 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:50.070196 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:50.071542 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:50.071545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:50.073223 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:50.073234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:50.074253 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:50.074257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:50.074510 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:50.075285 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:48:50.075309 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:48:50.077127 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:48:50.079403 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:48:50.079408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:48:50.079421 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:48:50.080095 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:48:50.080466 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:48:50.080469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:48:50.080470 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:48:50.080556 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:48:50.081149 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:48:50.081157 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:48:50.081385 (+ 228us) client_negotiation.cc:769] Sending connection context
0928 06:48:50.081562 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 06:48:50.081920 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":382,"spinlock_wait_cycles":31232,"thread_start_us":69,"threads_started":1}
I20260928 06:48:50.082115 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:50.067360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:38995 (local address 127.7.48.3:43811)
0928 06:48:50.067690 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0928 06:48:50.067705 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:48:50.067750 (+ 45us) server_negotiation.cc:407] Connection header received
0928 06:48:50.068475 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:48:50.068481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:48:50.068816 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:48:50.069121 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:48:50.070350 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:50.071325 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:50.073451 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:50.074120 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:50.074284 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:50.077637 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:48:50.077641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:48:50.077651 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:48:50.077678 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:48:50.079433 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:50.080150 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:50.080159 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:50.080167 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:50.080300 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:50.080669 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:50.080672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:50.080673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:50.081075 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:48:50.081224 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 06:48:50.081709 (+ 485us) server_negotiation.cc:299] Negotiation successful
0928 06:48:50.081956 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":136,"threads_started":1}
I20260928 06:48:50.082139 24358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:50.067192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:41411 (local address 127.7.48.2:45701)
0928 06:48:50.067450 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 06:48:50.067474 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 06:48:50.067795 (+ 321us) server_negotiation.cc:407] Connection header received
0928 06:48:50.068510 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:48:50.068515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:48:50.068816 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:48:50.069121 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:48:50.070350 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:50.071416 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:50.073364 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:48:50.074120 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:48:50.074284 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:50.077350 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:48:50.077356 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:48:50.077372 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:48:50.077409 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:48:50.079255 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:50.080208 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:50.080212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:50.080216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:50.080321 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:48:50.080661 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:48:50.080666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:48:50.080672 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:48:50.081030 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:48:50.081132 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 06:48:50.081705 (+ 573us) server_negotiation.cc:299] Negotiation successful
0928 06:48:50.081968 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":63,"threads_started":1}
I20260928 06:48:50.082137 24354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:50.067006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:43811 (local address 127.7.48.1:38995)
0928 06:48:50.067480 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:50.067497 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:48:50.067602 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:48:50.069163 (+ 1561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:48:50.069169 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:48:50.069691 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:48:50.070181 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:50.070196 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:50.071465 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:50.071469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:50.073223 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:48:50.073234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:48:50.074253 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:48:50.074257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:48:50.074508 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:48:50.075280 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:48:50.075307 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:48:50.077465 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:48:50.079537 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:48:50.079540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:48:50.079541 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:48:50.080046 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:48:50.080402 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:48:50.080409 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:48:50.080410 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:48:50.080552 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:48:50.081216 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:48:50.081218 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:48:50.081382 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 06:48:50.081562 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 06:48:50.081995 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":475,"spinlock_wait_cycles":9472}
W20260928 06:48:50.084882 23969 leader_election.cc:341] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20932349d3e147af931d4d553f6005db (127.7.48.2:45701): Not found: Tablet not found: eafa2e0eb1034cada1004f2e35d45a72
W20260928 06:48:50.085358 23970 leader_election.cc:341] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5c2c05c3b944b8b9d1f8889988ad296 (127.7.48.3:43811): Not found: Tablet not found: eafa2e0eb1034cada1004f2e35d45a72
I20260928 06:48:50.085460 23970 leader_election.cc:302] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 054e90aa690748eca63669817f350244; no voters: 20932349d3e147af931d4d553f6005db, b5c2c05c3b944b8b9d1f8889988ad296
I20260928 06:48:50.085758 24356 raft_consensus.cc:2726] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:48:50.102033 24360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:50.101708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.3:46361)
0928 06:48:50.101913 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:50.101957 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":74,"threads_started":1}
I20260928 06:48:50.821755 24361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:50.821369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.1:53067)
0928 06:48:50.821611 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:50.821662 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":72,"threads_started":1}
I20260928 06:48:50.922633 24362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:50.922201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.2:54789)
0928 06:48:50.922496 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:50.922545 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":107,"threads_started":1}
I20260928 06:48:51.103073 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:51.102619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.3:35973)
0928 06:48:51.102944 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:51.102996 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":72,"threads_started":1}
I20260928 06:48:51.729550 24364 raft_consensus.cc:482] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:48:51.729672 24364 raft_consensus.cc:504] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } }
I20260928 06:48:51.730155 24364 leader_election.cc:288] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20932349d3e147af931d4d553f6005db (127.7.48.2:45701), b5c2c05c3b944b8b9d1f8889988ad296 (127.7.48.3:43811)
W20260928 06:48:51.730997 23970 leader_election.cc:341] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5c2c05c3b944b8b9d1f8889988ad296 (127.7.48.3:43811): Not found: Tablet not found: eafa2e0eb1034cada1004f2e35d45a72
W20260928 06:48:51.731102 23969 leader_election.cc:341] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20932349d3e147af931d4d553f6005db (127.7.48.2:45701): Not found: Tablet not found: eafa2e0eb1034cada1004f2e35d45a72
I20260928 06:48:51.731310 23969 leader_election.cc:302] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 054e90aa690748eca63669817f350244; no voters: 20932349d3e147af931d4d553f6005db, b5c2c05c3b944b8b9d1f8889988ad296
I20260928 06:48:51.731484 24364 raft_consensus.cc:2726] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:48:51.823025 24365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:51.822624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.1:43115)
0928 06:48:51.822871 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:51.822930 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":77,"threads_started":1}
I20260928 06:48:51.924101 24366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:51.923594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.2:41249)
0928 06:48:51.923862 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:51.923980 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":79,"threads_started":1}
I20260928 06:48:52.104333 24367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:52.103956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.3:52475)
0928 06:48:52.104192 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:52.104244 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":69,"threads_started":1}
I20260928 06:48:52.824256 24368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:52.823800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.1:44801)
0928 06:48:52.824105 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:52.824162 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":131,"threads_started":1}
I20260928 06:48:52.925237 24369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:52.924826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.2:37331)
0928 06:48:52.925085 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:52.925145 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":91,"threads_started":1}
I20260928 06:48:53.105695 24370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:53.105137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.3:34159)
0928 06:48:53.105541 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:53.105604 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":86,"threads_started":1}
I20260928 06:48:53.825799 24371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:53.825318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.1:37247)
0928 06:48:53.825640 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:53.825704 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":77,"threads_started":1}
I20260928 06:48:53.926692 24372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:53.926235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.2:60065)
0928 06:48:53.926526 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:53.926594 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":82,"threads_started":1}
I20260928 06:48:54.107080 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:54.106636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.3:47301)
0928 06:48:54.106907 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:54.106961 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":97,"threads_started":1}
I20260928 06:48:54.230751 24374 raft_consensus.cc:482] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:48:54.230907 24374 raft_consensus.cc:504] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } }
I20260928 06:48:54.231420 24374 leader_election.cc:288] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20932349d3e147af931d4d553f6005db (127.7.48.2:45701), b5c2c05c3b944b8b9d1f8889988ad296 (127.7.48.3:43811)
W20260928 06:48:54.232188 23969 leader_election.cc:341] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20932349d3e147af931d4d553f6005db (127.7.48.2:45701): Not found: Tablet not found: eafa2e0eb1034cada1004f2e35d45a72
W20260928 06:48:54.232292 23970 leader_election.cc:341] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5c2c05c3b944b8b9d1f8889988ad296 (127.7.48.3:43811): Not found: Tablet not found: eafa2e0eb1034cada1004f2e35d45a72
I20260928 06:48:54.232414 23970 leader_election.cc:302] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 054e90aa690748eca63669817f350244; no voters: 20932349d3e147af931d4d553f6005db, b5c2c05c3b944b8b9d1f8889988ad296
I20260928 06:48:54.232582 24374 raft_consensus.cc:2726] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:48:54.827054 24375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:54.826609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.1:47279)
0928 06:48:54.826886 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:54.826959 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":73,"threads_started":1}
I20260928 06:48:54.928022 24376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:54.927593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.2:52865)
0928 06:48:54.927824 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:54.927917 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260928 06:48:55.108376 24377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:55.107969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.3:59169)
0928 06:48:55.108232 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:55.108287 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":72,"threads_started":1}
I20260928 06:48:55.828367 24378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:55.827863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.1:37183)
0928 06:48:55.828209 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:55.828265 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":124,"threads_started":1}
I20260928 06:48:55.929117 24379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:55.928688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.2:54851)
0928 06:48:55.928979 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:55.929031 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":92,"threads_started":1}
I20260928 06:48:56.109530 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:56.109097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.3:59853)
0928 06:48:56.109387 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:56.109440 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":89,"threads_started":1}
I20260928 06:48:56.829563 24381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:56.829110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.1:49671)
0928 06:48:56.829400 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:56.829464 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":95,"threads_started":1}
I20260928 06:48:56.930248 24382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:56.929850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.2:47699)
0928 06:48:56.930105 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:56.930156 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":94,"threads_started":1}
I20260928 06:48:57.110874 24383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:57.110412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.3:56659)
0928 06:48:57.110681 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:57.110752 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":108,"threads_started":1}
I20260928 06:48:57.280552 24384 raft_consensus.cc:482] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:48:57.280683 24384 raft_consensus.cc:504] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } }
I20260928 06:48:57.281164 24384 leader_election.cc:288] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20932349d3e147af931d4d553f6005db (127.7.48.2:45701), b5c2c05c3b944b8b9d1f8889988ad296 (127.7.48.3:43811)
W20260928 06:48:57.282063 23969 leader_election.cc:341] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20932349d3e147af931d4d553f6005db (127.7.48.2:45701): Not found: Tablet not found: eafa2e0eb1034cada1004f2e35d45a72
W20260928 06:48:57.282178 23970 leader_election.cc:341] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5c2c05c3b944b8b9d1f8889988ad296 (127.7.48.3:43811): Not found: Tablet not found: eafa2e0eb1034cada1004f2e35d45a72
I20260928 06:48:57.282225 23969 leader_election.cc:302] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 054e90aa690748eca63669817f350244; no voters: 20932349d3e147af931d4d553f6005db, b5c2c05c3b944b8b9d1f8889988ad296
I20260928 06:48:57.282356 24384 raft_consensus.cc:2726] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:48:57.830822 24385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:57.830326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.1:47383)
0928 06:48:57.830659 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:57.830721 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":124,"threads_started":1}
I20260928 06:48:57.931468 24386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:57.931064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.2:60381)
0928 06:48:57.931309 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:57.931367 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260928 06:48:58.112038 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:58.111617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.3:60545)
0928 06:48:58.111864 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:58.111942 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":78,"threads_started":1}
I20260928 06:48:58.832202 24388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:58.831743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.1:54567)
0928 06:48:58.832044 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:58.832106 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":76,"threads_started":1}
I20260928 06:48:58.932745 24389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:58.932295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.2:49909)
0928 06:48:58.932568 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:58.932630 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":76,"threads_started":1}
I20260928 06:48:59.113368 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:59.112883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.3:34471)
0928 06:48:59.113220 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:59.113272 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":125,"threads_started":1}
I20260928 06:48:59.833353 24391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:59.832928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.1:40911)
0928 06:48:59.833199 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:59.833257 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":98,"threads_started":1}
I20260928 06:48:59.934039 24392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:48:59.933587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.2:44975)
0928 06:48:59.933877 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 06:48:59.933939 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":95,"threads_started":1}
I20260928 06:49:00.059587 24393 raft_consensus.cc:482] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:49:00.059729 24393 raft_consensus.cc:504] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20932349d3e147af931d4d553f6005db" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 45701 } } peers { permanent_uuid: "b5c2c05c3b944b8b9d1f8889988ad296" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43811 } } peers { permanent_uuid: "054e90aa690748eca63669817f350244" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 40317 } }
I20260928 06:49:00.060262 24393 leader_election.cc:288] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20932349d3e147af931d4d553f6005db (127.7.48.2:45701), b5c2c05c3b944b8b9d1f8889988ad296 (127.7.48.3:43811)
W20260928 06:49:00.061091 23969 leader_election.cc:341] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20932349d3e147af931d4d553f6005db (127.7.48.2:45701): Not found: Tablet not found: eafa2e0eb1034cada1004f2e35d45a72
W20260928 06:49:00.061203 23970 leader_election.cc:341] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5c2c05c3b944b8b9d1f8889988ad296 (127.7.48.3:43811): Not found: Tablet not found: eafa2e0eb1034cada1004f2e35d45a72
I20260928 06:49:00.061314 23970 leader_election.cc:302] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 054e90aa690748eca63669817f350244; no voters: 20932349d3e147af931d4d553f6005db, b5c2c05c3b944b8b9d1f8889988ad296
I20260928 06:49:00.061517 24393 raft_consensus.cc:2726] T eafa2e0eb1034cada1004f2e35d45a72 P 054e90aa690748eca63669817f350244 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:49:00.114708 24394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:00.114250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.3:60591)
0928 06:49:00.114523 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:00.114598 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":79,"threads_started":1}
I20260928 06:49:00.834713 24395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:00.834275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.1:37631)
0928 06:49:00.834536 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:00.834617 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
I20260928 06:49:00.935209 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:00.934798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.2:48803)
0928 06:49:00.935052 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:00.935102 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":111,"threads_started":1}
I20260928 06:49:01.115868 24397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:01.115436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.3:40863)
0928 06:49:01.115702 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:01.115755 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":101,"threads_started":1}
I20260928 06:49:01.836162 24398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:01.835639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.1:57951)
0928 06:49:01.835948 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:01.836025 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":83,"threads_started":1}
I20260928 06:49:01.936533 24399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:01.936033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.2:59063)
0928 06:49:01.936361 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:01.936428 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":80,"threads_started":1}
I20260928 06:49:02.117121 24400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:02.116728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.3:53505)
0928 06:49:02.116965 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:02.117021 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260928 06:49:02.837464 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:02.836959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.1:56837)
0928 06:49:02.837242 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:02.837301 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":99,"threads_started":1}
I20260928 06:49:02.937788 24402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:02.937264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.2:41731)
0928 06:49:02.937597 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:02.937682 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":81,"threads_started":1}
I20260928 06:49:03.118373 24403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:03.117959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.3:41983)
0928 06:49:03.118225 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:03.118279 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":74,"threads_started":1}
I20260928 06:49:03.839021 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:03.838598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.1:42257)
0928 06:49:03.838856 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:03.838918 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":76,"threads_started":1}
I20260928 06:49:03.939239 24405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:03.938824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33141 (local address 127.7.48.2:35757)
0928 06:49:03.939081 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:03.939141 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.62:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":77,"threads_started":1}
I20260928 06:49:04.106810 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 23913
I20260928 06:49:04.113089 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 24087
I20260928 06:49:04.118809 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 24220
2026-09-28T06:49:04Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59061 ms)
[----------] 30 tests from SecurityITest (238586 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:49:04 dist-test-slave-twl6 krb5kdc[24408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:49:04 dist-test-slave-twl6 krb5kdc[24408](info): set up 2 sockets
Sep 28 06:49:04 dist-test-slave-twl6 krb5kdc[24408](info): commencing operation
krb5kdc: starting...
W20260928 06:49:05.152181 7360 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:49:05 dist-test-slave-twl6 krb5kdc[24408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:49:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:49:05.279043 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:34583
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:40741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:34583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:05.393242 24421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:05.393553 24421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:05.393603 24421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:05.397758 24421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:05.397817 24421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:05.397837 24421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:05.397897 24421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:05.397924 24421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:05.403270 24421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:40741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:34583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:34583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:05.404366 24421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:05.405454 24421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:49:05.414703 24421 server_base.cc:1034] running on GCE node
W20260928 06:49:05.414638 24426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:05.414757 24429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:05.414640 24427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:05.415428 24421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:05.416602 24421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:05.417768 24421 hybrid_clock.cc:648] HybridClock initialized: now 1790578145417749 us; error 31 us; skew 500 ppm
Sep 28 06:49:05 dist-test-slave-twl6 krb5kdc[24408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578145, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:05.420436 24421 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:49:05.421260 24421 webserver.cc:460] Webserver started at http://127.7.48.62:44987/ using document root <none> and password file <none>
I20260928 06:49:05.421820 24421 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:05.421913 24421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:05.422159 24421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:05.424239 24421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "743e5d900bee4de7808f1510c3850217"
format_stamp: "Formatted at 2026-09-28 06:49:05 on dist-test-slave-twl6"
server_key: "326faa55f41733477fa467488a8b79c7"
server_key_iv: "6f357f91a344d16e2d227899302e7e80"
server_key_version: "encryptionkey@0"
I20260928 06:49:05.424844 24421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "743e5d900bee4de7808f1510c3850217"
format_stamp: "Formatted at 2026-09-28 06:49:05 on dist-test-slave-twl6"
server_key: "326faa55f41733477fa467488a8b79c7"
server_key_iv: "6f357f91a344d16e2d227899302e7e80"
server_key_version: "encryptionkey@0"
I20260928 06:49:05.428498 24421 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 06:49:05.430831 24436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:05.432111 24421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 06:49:05.432243 24421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "743e5d900bee4de7808f1510c3850217"
format_stamp: "Formatted at 2026-09-28 06:49:05 on dist-test-slave-twl6"
server_key: "326faa55f41733477fa467488a8b79c7"
server_key_iv: "6f357f91a344d16e2d227899302e7e80"
server_key_version: "encryptionkey@0"
I20260928 06:49:05.432358 24421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:05.449512 24421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:05.452605 24421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:05.452811 24421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:05.459702 24421 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:34583
I20260928 06:49:05.459758 24488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:34583 every 8 connection(s)
I20260928 06:49:05.460736 24421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:05.463441 24489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:05.466053 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 24421
I20260928 06:49:05.466221 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:05.466511 7360 external_mini_cluster.cc:1428] Setting key 1845807fde3d196d558e4d62a0a153ed
I20260928 06:49:05.470237 24489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 743e5d900bee4de7808f1510c3850217: Bootstrap starting.
I20260928 06:49:05.472877 24489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 743e5d900bee4de7808f1510c3850217: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:05.473786 24489 log.cc:824] T 00000000000000000000000000000000 P 743e5d900bee4de7808f1510c3850217: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:05.475978 24489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 743e5d900bee4de7808f1510c3850217: No bootstrap required, opened a new log
Sep 28 06:49:05 dist-test-slave-twl6 krb5kdc[24408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:49:05.480693 24489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 743e5d900bee4de7808f1510c3850217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743e5d900bee4de7808f1510c3850217" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34583 } }
I20260928 06:49:05.481012 24489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 743e5d900bee4de7808f1510c3850217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:05.481084 24489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 743e5d900bee4de7808f1510c3850217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 743e5d900bee4de7808f1510c3850217, State: Initialized, Role: FOLLOWER
I20260928 06:49:05.481566 24489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 743e5d900bee4de7808f1510c3850217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743e5d900bee4de7808f1510c3850217" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34583 } }
I20260928 06:49:05.481724 24489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 743e5d900bee4de7808f1510c3850217 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:49:05.481791 24489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 743e5d900bee4de7808f1510c3850217 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:49:05.481891 24489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 743e5d900bee4de7808f1510c3850217 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:05.481906 24492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:05.467972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42198 (local address 127.7.48.62:34583)
0928 06:49:05.468415 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:05.468435 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:05.468478 (+ 43us) server_negotiation.cc:407] Connection header received
0928 06:49:05.469300 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:05.469325 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:05.469733 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:05.470100 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:05.470839 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:05.472352 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:05.473257 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:05.473959 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:05.474143 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:05.476860 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:05.476865 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:05.476878 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:05.476904 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:05.478568 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:05.479065 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:05.479070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:05.479074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:05.479185 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:05.479488 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:05.479493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:05.479496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:05.480123 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:05.480373 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:05.480703 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 06:49:05.481024 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":111,"threads_started":1}
I20260928 06:49:05.483108 24489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 743e5d900bee4de7808f1510c3850217 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743e5d900bee4de7808f1510c3850217" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34583 } }
I20260928 06:49:05.483453 24489 leader_election.cc:302] T 00000000000000000000000000000000 P 743e5d900bee4de7808f1510c3850217 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 743e5d900bee4de7808f1510c3850217; no voters:
I20260928 06:49:05.483767 24489 leader_election.cc:288] T 00000000000000000000000000000000 P 743e5d900bee4de7808f1510c3850217 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:49:05.483901 24494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 743e5d900bee4de7808f1510c3850217 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:05.484130 24494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 743e5d900bee4de7808f1510c3850217 [term 1 LEADER]: Becoming Leader. State: Replica: 743e5d900bee4de7808f1510c3850217, State: Running, Role: LEADER
I20260928 06:49:05.484513 24494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 743e5d900bee4de7808f1510c3850217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743e5d900bee4de7808f1510c3850217" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34583 } }
I20260928 06:49:05.485014 24489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 743e5d900bee4de7808f1510c3850217 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:05.485661 24496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 743e5d900bee4de7808f1510c3850217 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 743e5d900bee4de7808f1510c3850217. Latest consensus state: current_term: 1 leader_uuid: "743e5d900bee4de7808f1510c3850217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743e5d900bee4de7808f1510c3850217" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34583 } } }
I20260928 06:49:05.485791 24496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 743e5d900bee4de7808f1510c3850217 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:05.486156 24495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 743e5d900bee4de7808f1510c3850217 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "743e5d900bee4de7808f1510c3850217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743e5d900bee4de7808f1510c3850217" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34583 } } }
I20260928 06:49:05.486340 24495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 743e5d900bee4de7808f1510c3850217 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:05.486580 24500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:05.489842 24500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:49:05.495396 24500 catalog_manager.cc:1347] Generated new cluster ID: 8ea62eefe3e048af927b9436eec8d5bb
I20260928 06:49:05.495462 24500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:05.509749 24500 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:05.510715 24500 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:05.525177 24500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 743e5d900bee4de7808f1510c3850217: Generated new TSK 0
I20260928 06:49:05.525919 24500 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
I20260928 06:49:05.559417 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:34583
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:40741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:05.709364 24516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:05.709666 24516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:05.709709 24516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:05.713747 24516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:05.713819 24516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:05.713878 24516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:49:05.719110 24516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:40741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:34583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:05.720208 24516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:05.721175 24516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:05.730777 24521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:05.730794 24524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:05.730783 24522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:05.731158 24516 server_base.cc:1034] running on GCE node
I20260928 06:49:05.731665 24516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:05.732393 24516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:05.733584 24516 hybrid_clock.cc:648] HybridClock initialized: now 1790578145733543 us; error 49 us; skew 500 ppm
Sep 28 06:49:05 dist-test-slave-twl6 krb5kdc[24408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578145, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:05.736527 24516 init.cc:377] Logged in from keytab as kudu/127.7.48.1@KRBTEST.COM (short username kudu)
I20260928 06:49:05.737468 24516 webserver.cc:460] Webserver started at http://127.7.48.1:38427/ using document root <none> and password file <none>
I20260928 06:49:05.738068 24516 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:05.738161 24516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:05.738394 24516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:05.740518 24516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "28ecb9a2c8104db4a4751cbe5c094c35"
format_stamp: "Formatted at 2026-09-28 06:49:05 on dist-test-slave-twl6"
server_key: "7a93a57a41ac63bf62af602d5e2e3716"
server_key_iv: "b4bc20e0737853eb47011e2d12789347"
server_key_version: "encryptionkey@0"
I20260928 06:49:05.741089 24516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "28ecb9a2c8104db4a4751cbe5c094c35"
format_stamp: "Formatted at 2026-09-28 06:49:05 on dist-test-slave-twl6"
server_key: "7a93a57a41ac63bf62af602d5e2e3716"
server_key_iv: "b4bc20e0737853eb47011e2d12789347"
server_key_version: "encryptionkey@0"
I20260928 06:49:05.744971 24516 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 06:49:05.747787 24531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:05.749099 24516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:49:05.749251 24516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "28ecb9a2c8104db4a4751cbe5c094c35"
format_stamp: "Formatted at 2026-09-28 06:49:05 on dist-test-slave-twl6"
server_key: "7a93a57a41ac63bf62af602d5e2e3716"
server_key_iv: "b4bc20e0737853eb47011e2d12789347"
server_key_version: "encryptionkey@0"
I20260928 06:49:05.749377 24516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:05.778673 24516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:05.781927 24516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:05.782159 24516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:05.782790 24516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:49:05.783649 24516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:49:05.783723 24516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:05.783797 24516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:49:05.783841 24516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:05.793617 24516 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:41747
I20260928 06:49:05.793641 24644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:41747 every 8 connection(s)
I20260928 06:49:05.794550 24516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:49:05.797091 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 24516
I20260928 06:49:05.797236 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
I20260928 06:49:05.797552 7360 external_mini_cluster.cc:1428] Setting key 50b98f506b86499548854a0774041d3c
Sep 28 06:49:05 dist-test-slave-twl6 krb5kdc[24408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578145, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.48.2@KRBTEST.COM; defaulting to no policy
I20260928 06:49:05.810489 24492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:05.796491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:34875 (local address 127.7.48.62:34583)
0928 06:49:05.796663 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:05.796666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:05.797716 (+ 1050us) server_negotiation.cc:407] Connection header received
0928 06:49:05.799495 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:05.799500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:05.799554 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:05.799647 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:05.801154 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:05.801709 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:05.803556 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:05.803790 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:05.803871 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:05.806996 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:05.806999 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:05.807001 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:05.807025 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:05.808528 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:05.809046 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:05.809049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:05.809050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:05.809126 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:05.809421 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:05.809423 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:05.809425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:05.809606 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:05.809681 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:05.810164 (+ 483us) server_negotiation.cc:299] Negotiation successful
0928 06:49:05.810271 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.2@KRBTEST.COM" created.
I20260928 06:49:05.811396 24647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:05.796946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34583 (local address 127.7.48.1:34875)
0928 06:49:05.797535 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:05.797581 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:05.799273 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:05.799841 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:05.799849 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:05.800273 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:05.800973 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:05.800998 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:05.801847 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:05.801851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:05.803425 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:05.803441 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:05.803990 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:05.803997 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:05.804260 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:05.804934 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:05.804960 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:05.806817 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:05.808639 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:05.808644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:05.808658 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:05.808957 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:05.809205 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:05.809207 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:05.809209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:05.809343 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:05.809695 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:05.809701 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:05.809942 (+ 241us) client_negotiation.cc:769] Sending connection context
0928 06:49:05.810116 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 06:49:05.810382 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":157,"threads_started":1}
I20260928 06:49:05.812893 24645 heartbeater.cc:347] Connected to a master server at 127.7.48.62:34583
I20260928 06:49:05.813155 24645 heartbeater.cc:464] Registering TS with master...
I20260928 06:49:05.813676 24645 heartbeater.cc:511] Master 127.7.48.62:34583 requested a full tablet report, sending...
I20260928 06:49:05.815441 24453 ts_manager.cc:194] Registered new tserver with Master: 28ecb9a2c8104db4a4751cbe5c094c35 (127.7.48.1:41747)
I20260928 06:49:05.817243 24453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.1@KRBTEST.COM'} at 127.7.48.1:34875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
I20260928 06:49:05.827214 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:34583
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:40741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:05.943128 24650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:05.943423 24650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:05.943511 24650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:05.947814 24650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:05.947935 24650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:05.948081 24650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:49:05.953560 24650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:40741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:34583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:05.954687 24650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:05.955741 24650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:05.965389 24655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:05.965437 24656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:05.965544 24650 server_base.cc:1034] running on GCE node
W20260928 06:49:05.965401 24658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:05.966096 24650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:05.966825 24650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:05.967957 24650 hybrid_clock.cc:648] HybridClock initialized: now 1790578145967936 us; error 29 us; skew 500 ppm
Sep 28 06:49:05 dist-test-slave-twl6 krb5kdc[24408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578145, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:05.970726 24650 init.cc:377] Logged in from keytab as kudu/127.7.48.2@KRBTEST.COM (short username kudu)
I20260928 06:49:05.971664 24650 webserver.cc:460] Webserver started at http://127.7.48.2:37897/ using document root <none> and password file <none>
I20260928 06:49:05.972319 24650 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:05.972412 24650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:05.972645 24650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:05.974857 24650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "1a30b21ab77a4f40b81d1e0910257f83"
format_stamp: "Formatted at 2026-09-28 06:49:05 on dist-test-slave-twl6"
server_key: "fd159873ee2579e5af1023124755d95c"
server_key_iv: "9dae9e4cb460bd6ba415cd257e9c915c"
server_key_version: "encryptionkey@0"
I20260928 06:49:05.975431 24650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "1a30b21ab77a4f40b81d1e0910257f83"
format_stamp: "Formatted at 2026-09-28 06:49:05 on dist-test-slave-twl6"
server_key: "fd159873ee2579e5af1023124755d95c"
server_key_iv: "9dae9e4cb460bd6ba415cd257e9c915c"
server_key_version: "encryptionkey@0"
I20260928 06:49:05.979228 24650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 06:49:05.981796 24665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:05.983013 24650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 06:49:05.983167 24650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "1a30b21ab77a4f40b81d1e0910257f83"
format_stamp: "Formatted at 2026-09-28 06:49:05 on dist-test-slave-twl6"
server_key: "fd159873ee2579e5af1023124755d95c"
server_key_iv: "9dae9e4cb460bd6ba415cd257e9c915c"
server_key_version: "encryptionkey@0"
I20260928 06:49:05.983287 24650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:05.996301 24650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:05.999086 24650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:05.999298 24650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:05.999951 24650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:49:06.000792 24650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:49:06.000859 24650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:06.000924 24650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:49:06.000970 24650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:06.009922 24650 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:37571
I20260928 06:49:06.009943 24778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:37571 every 8 connection(s)
I20260928 06:49:06.010807 24650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:49:06.014294 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 24650
I20260928 06:49:06.014447 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:49:06.014732 7360 external_mini_cluster.cc:1428] Setting key d73fb259c40f53cf853a09386d7ff376
Sep 28 06:49:06 dist-test-slave-twl6 krb5kdc[24408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578145, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.48.3@KRBTEST.COM; defaulting to no policy
I20260928 06:49:06.025810 24492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:06.012734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:48327 (local address 127.7.48.62:34583)
0928 06:49:06.012875 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:06.012879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:06.013734 (+ 855us) server_negotiation.cc:407] Connection header received
0928 06:49:06.015237 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:06.015240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:06.015299 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:06.015416 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:06.016663 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.017186 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.018341 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.018552 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.018654 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:06.021723 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:06.021727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:06.021729 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:06.021754 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:06.023313 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:06.023988 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:06.023990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:06.023991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:06.024064 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:06.024536 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:06.024545 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:06.024550 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:06.024763 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:06.024842 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:06.025490 (+ 648us) server_negotiation.cc:299] Negotiation successful
0928 06:49:06.025627 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.3@KRBTEST.COM" created.
I20260928 06:49:06.026822 24781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:06.013128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34583 (local address 127.7.48.2:48327)
0928 06:49:06.013584 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:06.013627 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:06.015022 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:06.015592 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:06.015599 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:06.016016 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:06.016515 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:06.016529 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.017316 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.017323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:06.018204 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:06.018213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.018709 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.018713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:06.018874 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:06.019520 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:06.019536 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:06.021525 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:06.023444 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:06.023450 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:06.023471 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:06.023846 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:06.024200 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:06.024206 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:06.024209 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:06.024412 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:06.024864 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:06.024869 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:06.025200 (+ 331us) client_negotiation.cc:769] Sending connection context
0928 06:49:06.025417 (+ 217us) client_negotiation.cc:240] Negotiation successful
0928 06:49:06.025768 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":119,"threads_started":1}
I20260928 06:49:06.028157 24779 heartbeater.cc:347] Connected to a master server at 127.7.48.62:34583
I20260928 06:49:06.028442 24779 heartbeater.cc:464] Registering TS with master...
I20260928 06:49:06.028964 24779 heartbeater.cc:511] Master 127.7.48.62:34583 requested a full tablet report, sending...
I20260928 06:49:06.030066 24453 ts_manager.cc:194] Registered new tserver with Master: 1a30b21ab77a4f40b81d1e0910257f83 (127.7.48.2:37571)
I20260928 06:49:06.030613 24453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.2@KRBTEST.COM'} at 127.7.48.2:48327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
I20260928 06:49:06.043851 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:34583
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:40741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:06.157465 24784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:06.157799 24784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:06.157883 24784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:06.162072 24784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:06.162168 24784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:06.162262 24784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:49:06.167742 24784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:40741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:34583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:06.169032 24784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:06.170136 24784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:06.180374 24789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:06.180358 24790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:06.180488 24784 server_base.cc:1034] running on GCE node
W20260928 06:49:06.180379 24792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:06.181118 24784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:06.181751 24784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:06.182912 24784 hybrid_clock.cc:648] HybridClock initialized: now 1790578146182879 us; error 48 us; skew 500 ppm
Sep 28 06:49:06 dist-test-slave-twl6 krb5kdc[24408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578146, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:06.190135 24784 init.cc:377] Logged in from keytab as kudu/127.7.48.3@KRBTEST.COM (short username kudu)
I20260928 06:49:06.191200 24784 webserver.cc:460] Webserver started at http://127.7.48.3:46837/ using document root <none> and password file <none>
I20260928 06:49:06.191841 24784 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:06.191998 24784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:06.192258 24784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:06.194455 24784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "14ffbd8aab1646c395c9a689eb6a8ddc"
format_stamp: "Formatted at 2026-09-28 06:49:06 on dist-test-slave-twl6"
server_key: "f4b524cac347c97369c3f1b7605a07ca"
server_key_iv: "a455a4e2fd33883f14d37fa3d643b261"
server_key_version: "encryptionkey@0"
I20260928 06:49:06.195041 24784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "14ffbd8aab1646c395c9a689eb6a8ddc"
format_stamp: "Formatted at 2026-09-28 06:49:06 on dist-test-slave-twl6"
server_key: "f4b524cac347c97369c3f1b7605a07ca"
server_key_iv: "a455a4e2fd33883f14d37fa3d643b261"
server_key_version: "encryptionkey@0"
I20260928 06:49:06.199424 24784 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 06:49:06.202138 24799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:06.203369 24784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:49:06.203514 24784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "14ffbd8aab1646c395c9a689eb6a8ddc"
format_stamp: "Formatted at 2026-09-28 06:49:06 on dist-test-slave-twl6"
server_key: "f4b524cac347c97369c3f1b7605a07ca"
server_key_iv: "a455a4e2fd33883f14d37fa3d643b261"
server_key_version: "encryptionkey@0"
I20260928 06:49:06.203652 24784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:06.216128 24784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:06.219301 24784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:06.219535 24784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:06.220294 24784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:49:06.221210 24784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:49:06.221282 24784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:06.221356 24784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:49:06.221410 24784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:06.230752 24784 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:43877
I20260928 06:49:06.230782 24912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:43877 every 8 connection(s)
I20260928 06:49:06.231731 24784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:49:06.240715 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 24784
I20260928 06:49:06.240865 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:49:06.241196 7360 external_mini_cluster.cc:1428] Setting key de9f0ee0e96de35943e9db9d4a702de0
Sep 28 06:49:06 dist-test-slave-twl6 krb5kdc[24408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578146, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:49:06.246186 24492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:06.233728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:50905 (local address 127.7.48.62:34583)
0928 06:49:06.233876 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:06.233886 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:06.234767 (+ 881us) server_negotiation.cc:407] Connection header received
0928 06:49:06.236249 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:06.236251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:06.236310 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:06.236389 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:06.237667 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.238208 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.239334 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.239545 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.239623 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:06.242544 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:06.242548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:06.242551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:06.242583 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:06.244232 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:06.244730 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:06.244732 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:06.244733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:06.244809 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:06.245195 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:06.245197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:06.245198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:06.245386 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:06.245463 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:06.245948 (+ 485us) server_negotiation.cc:299] Negotiation successful
0928 06:49:06.246050 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:49:06.247107 24915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:06.234129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34583 (local address 127.7.48.3:50905)
0928 06:49:06.234600 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:06.234645 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:06.236027 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:06.236561 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:06.236569 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:06.237006 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:06.237518 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:06.237533 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.238322 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.238324 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:06.239226 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:06.239234 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.239688 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.239691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:06.239837 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:06.240514 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:06.240531 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:06.242351 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:06.244343 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:06.244348 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:06.244362 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:06.244642 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:06.244917 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:06.244922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:06.244926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:06.245108 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:06.245477 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:06.245482 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:06.245736 (+ 254us) client_negotiation.cc:769] Sending connection context
0928 06:49:06.245901 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 06:49:06.246184 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":123,"threads_started":1}
I20260928 06:49:06.248333 24913 heartbeater.cc:347] Connected to a master server at 127.7.48.62:34583
I20260928 06:49:06.248589 24913 heartbeater.cc:464] Registering TS with master...
I20260928 06:49:06.249128 24913 heartbeater.cc:511] Master 127.7.48.62:34583 requested a full tablet report, sending...
I20260928 06:49:06.250205 24453 ts_manager.cc:194] Registered new tserver with Master: 14ffbd8aab1646c395c9a689eb6a8ddc (127.7.48.3:43877)
I20260928 06:49:06.250793 24453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.3@KRBTEST.COM'} at 127.7.48.3:50905
I20260928 06:49:06.255558 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:49:06.265151 24492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:06.257633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42204 (local address 127.7.48.62:34583)
0928 06:49:06.257799 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:06.257810 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:06.257934 (+ 124us) server_negotiation.cc:407] Connection header received
0928 06:49:06.258040 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:06.258042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:06.258089 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:06.258156 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:06.258874 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.259383 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.260296 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.260503 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.260597 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:06.261998 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:06.262001 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:06.262004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:06.262027 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:06.263493 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:06.263959 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:06.263962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:06.263963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:06.264038 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:06.264286 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:06.264288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:06.264289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:06.264508 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:06.264628 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:06.264820 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 06:49:06.264920 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 06:49:06.268276 24453 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:49:06.271448 24453 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:49:06.288007 24926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:06.280730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:41747 (local address 127.0.0.1:56626)
0928 06:49:06.281830 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:06.281889 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:06.282053 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:06.283498 (+ 1445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:06.283500 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:06.283528 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:06.283618 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:06.283624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.284735 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.284738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:06.286310 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:06.286321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.287456 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.287459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:06.287546 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:06.287567 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 06:49:06.287649 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 06:49:06.287801 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"mutex_wait_us":109,"spinlock_wait_cycles":6656,"thread_start_us":104,"threads_started":1}
I20260928 06:49:06.288008 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:06.281985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33798 (local address 127.7.48.3:43877)
0928 06:49:06.282535 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:06.282546 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:06.282565 (+ 19us) server_negotiation.cc:407] Connection header received
0928 06:49:06.282627 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:06.282630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:06.282815 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:06.282972 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:06.283377 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.284519 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.286397 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.287264 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.287391 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:06.287591 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:06.287711 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 06:49:06.287811 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":163,"threads_started":1}
I20260928 06:49:06.288043 24924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:06.280755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:37571 (local address 127.0.0.1:41832)
0928 06:49:06.281830 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:06.281889 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:06.282053 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:06.282872 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:06.282878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:06.282902 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:06.283110 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:06.283117 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.284672 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.284677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:06.286414 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:06.286424 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.287653 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.287657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:06.287749 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:06.287769 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 06:49:06.287833 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 06:49:06.287925 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"mutex_wait_us":314,"spinlock_wait_cycles":2304,"thread_start_us":86,"threads_started":1}
I20260928 06:49:06.288064 24925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:06.280979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:43877 (local address 127.0.0.1:33798)
0928 06:49:06.282304 (+ 1325us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:06.282327 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:06.282421 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:06.282948 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:06.282951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:06.282970 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:06.283253 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:06.283260 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.284948 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.284952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:06.286281 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:06.286291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.287392 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.287396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:06.287502 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:06.287556 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 06:49:06.287680 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 06:49:06.287832 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1258,"mutex_wait_us":226,"thread_start_us":112,"threads_started":1}
I20260928 06:49:06.288645 24927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:06.281009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41832 (local address 127.7.48.2:37571)
0928 06:49:06.281701 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:06.281714 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:06.281993 (+ 279us) server_negotiation.cc:407] Connection header received
0928 06:49:06.282241 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:06.282247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:06.282699 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:06.282880 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:06.283231 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.284519 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.286535 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.287327 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.287429 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:06.287591 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:06.288376 (+ 785us) server_negotiation.cc:299] Negotiation successful
0928 06:49:06.288480 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":152,"threads_started":1}
I20260928 06:49:06.288875 24928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:06.281594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56626 (local address 127.7.48.1:41747)
0928 06:49:06.281892 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:06.281918 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:06.282044 (+ 126us) server_negotiation.cc:407] Connection header received
0928 06:49:06.283117 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:06.283126 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:06.283327 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:06.283511 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:06.283743 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.284582 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.286457 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.287308 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.288418 (+ 1110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:06.288553 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:06.288613 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 06:49:06.288716 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":123,"threads_started":1}
I20260928 06:49:06.291373 24847 tablet_service.cc:1467] Processing CreateTablet for tablet be7241cb56de4e36a5adacd9efa022c6 (DEFAULT_TABLE table=test-table [id=9d3ad394d17c4bc7beaa300065efdf48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:49:06.291374 24713 tablet_service.cc:1467] Processing CreateTablet for tablet be7241cb56de4e36a5adacd9efa022c6 (DEFAULT_TABLE table=test-table [id=9d3ad394d17c4bc7beaa300065efdf48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:49:06.292483 24713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be7241cb56de4e36a5adacd9efa022c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:06.292495 24847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be7241cb56de4e36a5adacd9efa022c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:06.298031 24579 tablet_service.cc:1467] Processing CreateTablet for tablet be7241cb56de4e36a5adacd9efa022c6 (DEFAULT_TABLE table=test-table [id=9d3ad394d17c4bc7beaa300065efdf48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:49:06.298578 24930 tablet_bootstrap.cc:492] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc: Bootstrap starting.
I20260928 06:49:06.299190 24579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be7241cb56de4e36a5adacd9efa022c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:06.299959 24931 tablet_bootstrap.cc:492] T be7241cb56de4e36a5adacd9efa022c6 P 1a30b21ab77a4f40b81d1e0910257f83: Bootstrap starting.
I20260928 06:49:06.300962 24930 tablet_bootstrap.cc:654] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:06.302127 24931 tablet_bootstrap.cc:654] T be7241cb56de4e36a5adacd9efa022c6 P 1a30b21ab77a4f40b81d1e0910257f83: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:06.302142 24930 log.cc:824] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:06.303367 24931 log.cc:824] T be7241cb56de4e36a5adacd9efa022c6 P 1a30b21ab77a4f40b81d1e0910257f83: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:06.304163 24930 tablet_bootstrap.cc:492] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc: No bootstrap required, opened a new log
I20260928 06:49:06.304371 24930 ts_tablet_manager.cc:1397] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260928 06:49:06.305450 24931 tablet_bootstrap.cc:492] T be7241cb56de4e36a5adacd9efa022c6 P 1a30b21ab77a4f40b81d1e0910257f83: No bootstrap required, opened a new log
I20260928 06:49:06.305662 24931 ts_tablet_manager.cc:1397] T be7241cb56de4e36a5adacd9efa022c6 P 1a30b21ab77a4f40b81d1e0910257f83: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 06:49:06.307495 24934 tablet_bootstrap.cc:492] T be7241cb56de4e36a5adacd9efa022c6 P 28ecb9a2c8104db4a4751cbe5c094c35: Bootstrap starting.
I20260928 06:49:06.307933 24930 raft_consensus.cc:348] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ecb9a2c8104db4a4751cbe5c094c35" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 41747 } } peers { permanent_uuid: "1a30b21ab77a4f40b81d1e0910257f83" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 37571 } } peers { permanent_uuid: "14ffbd8aab1646c395c9a689eb6a8ddc" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43877 } }
I20260928 06:49:06.308199 24930 raft_consensus.cc:374] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:06.308254 24930 raft_consensus.cc:729] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ffbd8aab1646c395c9a689eb6a8ddc, State: Initialized, Role: FOLLOWER
I20260928 06:49:06.308735 24930 consensus_queue.cc:260] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ecb9a2c8104db4a4751cbe5c094c35" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 41747 } } peers { permanent_uuid: "1a30b21ab77a4f40b81d1e0910257f83" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 37571 } } peers { permanent_uuid: "14ffbd8aab1646c395c9a689eb6a8ddc" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43877 } }
I20260928 06:49:06.309379 24913 heartbeater.cc:503] Master 127.7.48.62:34583 was elected leader, sending a full tablet report...
I20260928 06:49:06.309835 24934 tablet_bootstrap.cc:654] T be7241cb56de4e36a5adacd9efa022c6 P 28ecb9a2c8104db4a4751cbe5c094c35: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:06.309918 24931 raft_consensus.cc:348] T be7241cb56de4e36a5adacd9efa022c6 P 1a30b21ab77a4f40b81d1e0910257f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ecb9a2c8104db4a4751cbe5c094c35" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 41747 } } peers { permanent_uuid: "1a30b21ab77a4f40b81d1e0910257f83" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 37571 } } peers { permanent_uuid: "14ffbd8aab1646c395c9a689eb6a8ddc" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43877 } }
I20260928 06:49:06.310220 24931 raft_consensus.cc:374] T be7241cb56de4e36a5adacd9efa022c6 P 1a30b21ab77a4f40b81d1e0910257f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:06.310297 24931 raft_consensus.cc:729] T be7241cb56de4e36a5adacd9efa022c6 P 1a30b21ab77a4f40b81d1e0910257f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a30b21ab77a4f40b81d1e0910257f83, State: Initialized, Role: FOLLOWER
I20260928 06:49:06.310966 24930 ts_tablet_manager.cc:1428] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260928 06:49:06.311007 24931 consensus_queue.cc:260] T be7241cb56de4e36a5adacd9efa022c6 P 1a30b21ab77a4f40b81d1e0910257f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ecb9a2c8104db4a4751cbe5c094c35" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 41747 } } peers { permanent_uuid: "1a30b21ab77a4f40b81d1e0910257f83" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 37571 } } peers { permanent_uuid: "14ffbd8aab1646c395c9a689eb6a8ddc" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43877 } }
I20260928 06:49:06.311151 24934 log.cc:824] T be7241cb56de4e36a5adacd9efa022c6 P 28ecb9a2c8104db4a4751cbe5c094c35: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:06.311692 24779 heartbeater.cc:503] Master 127.7.48.62:34583 was elected leader, sending a full tablet report...
I20260928 06:49:06.312014 24931 ts_tablet_manager.cc:1428] T be7241cb56de4e36a5adacd9efa022c6 P 1a30b21ab77a4f40b81d1e0910257f83: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260928 06:49:06.314103 24934 tablet_bootstrap.cc:492] T be7241cb56de4e36a5adacd9efa022c6 P 28ecb9a2c8104db4a4751cbe5c094c35: No bootstrap required, opened a new log
I20260928 06:49:06.314316 24934 ts_tablet_manager.cc:1397] T be7241cb56de4e36a5adacd9efa022c6 P 28ecb9a2c8104db4a4751cbe5c094c35: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 06:49:06.317463 24934 raft_consensus.cc:348] T be7241cb56de4e36a5adacd9efa022c6 P 28ecb9a2c8104db4a4751cbe5c094c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ecb9a2c8104db4a4751cbe5c094c35" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 41747 } } peers { permanent_uuid: "1a30b21ab77a4f40b81d1e0910257f83" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 37571 } } peers { permanent_uuid: "14ffbd8aab1646c395c9a689eb6a8ddc" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43877 } }
I20260928 06:49:06.317713 24934 raft_consensus.cc:374] T be7241cb56de4e36a5adacd9efa022c6 P 28ecb9a2c8104db4a4751cbe5c094c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:06.317798 24934 raft_consensus.cc:729] T be7241cb56de4e36a5adacd9efa022c6 P 28ecb9a2c8104db4a4751cbe5c094c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28ecb9a2c8104db4a4751cbe5c094c35, State: Initialized, Role: FOLLOWER
I20260928 06:49:06.318208 24934 consensus_queue.cc:260] T be7241cb56de4e36a5adacd9efa022c6 P 28ecb9a2c8104db4a4751cbe5c094c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ecb9a2c8104db4a4751cbe5c094c35" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 41747 } } peers { permanent_uuid: "1a30b21ab77a4f40b81d1e0910257f83" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 37571 } } peers { permanent_uuid: "14ffbd8aab1646c395c9a689eb6a8ddc" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43877 } }
I20260928 06:49:06.318835 24645 heartbeater.cc:503] Master 127.7.48.62:34583 was elected leader, sending a full tablet report...
I20260928 06:49:06.319149 24934 ts_tablet_manager.cc:1428] T be7241cb56de4e36a5adacd9efa022c6 P 28ecb9a2c8104db4a4751cbe5c094c35: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 06:49:06.324018 24935 raft_consensus.cc:482] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:49:06.324222 24935 raft_consensus.cc:504] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ecb9a2c8104db4a4751cbe5c094c35" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 41747 } } peers { permanent_uuid: "1a30b21ab77a4f40b81d1e0910257f83" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 37571 } } peers { permanent_uuid: "14ffbd8aab1646c395c9a689eb6a8ddc" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43877 } }
I20260928 06:49:06.325289 24935 leader_election.cc:288] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28ecb9a2c8104db4a4751cbe5c094c35 (127.7.48.1:41747), 1a30b21ab77a4f40b81d1e0910257f83 (127.7.48.2:37571)
I20260928 06:49:06.328979 24915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:06.325470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:41747 (local address 127.7.48.3:46159)
0928 06:49:06.325674 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:06.325690 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:06.325783 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:06.326053 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:06.326056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:06.326073 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:06.326193 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:06.326198 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.326937 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.326940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:06.327762 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:06.327775 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.328565 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.328576 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:06.328684 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:06.328712 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 06:49:06.328789 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 06:49:06.328862 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260928 06:49:06.329038 24928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:06.325577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:46159 (local address 127.7.48.1:41747)
0928 06:49:06.325676 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:06.325682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:06.325852 (+ 170us) server_negotiation.cc:407] Connection header received
0928 06:49:06.325902 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:06.325904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:06.325952 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:06.326034 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:06.326300 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.326819 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.327933 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.328456 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.328535 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:06.328586 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:06.328820 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 06:49:06.328891 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:49:06.329237 24939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:06.325559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:37571 (local address 127.7.48.3:43681)
0928 06:49:06.325895 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:06.325907 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:06.326038 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:06.326297 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:06.326300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:06.326316 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:06.326433 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:06.326439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.327152 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.327157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:06.328083 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:06.328093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.328832 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.328836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:06.328930 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:06.328952 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 06:49:06.329029 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 06:49:06.329100 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":107,"threads_started":1}
I20260928 06:49:06.329283 24927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:06.325672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:43681 (local address 127.7.48.2:37571)
0928 06:49:06.325807 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:06.325813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:06.325992 (+ 179us) server_negotiation.cc:407] Connection header received
0928 06:49:06.326148 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:06.326151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:06.326203 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:06.326286 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:06.326548 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.327039 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.328206 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.328688 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.328798 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:06.328852 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:06.329064 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 06:49:06.329158 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:49:06.329625 24599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7241cb56de4e36a5adacd9efa022c6" candidate_uuid: "14ffbd8aab1646c395c9a689eb6a8ddc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28ecb9a2c8104db4a4751cbe5c094c35" is_pre_election: true
I20260928 06:49:06.329960 24599 raft_consensus.cc:2454] T be7241cb56de4e36a5adacd9efa022c6 P 28ecb9a2c8104db4a4751cbe5c094c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14ffbd8aab1646c395c9a689eb6a8ddc in term 0.
I20260928 06:49:06.330015 24733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7241cb56de4e36a5adacd9efa022c6" candidate_uuid: "14ffbd8aab1646c395c9a689eb6a8ddc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a30b21ab77a4f40b81d1e0910257f83" is_pre_election: true
I20260928 06:49:06.330361 24733 raft_consensus.cc:2454] T be7241cb56de4e36a5adacd9efa022c6 P 1a30b21ab77a4f40b81d1e0910257f83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14ffbd8aab1646c395c9a689eb6a8ddc in term 0.
I20260928 06:49:06.330513 24800 leader_election.cc:302] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ffbd8aab1646c395c9a689eb6a8ddc, 28ecb9a2c8104db4a4751cbe5c094c35; no voters:
I20260928 06:49:06.330799 24935 raft_consensus.cc:2781] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:49:06.330888 24935 raft_consensus.cc:482] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:49:06.330920 24935 raft_consensus.cc:3037] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:06.332048 24935 raft_consensus.cc:504] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ecb9a2c8104db4a4751cbe5c094c35" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 41747 } } peers { permanent_uuid: "1a30b21ab77a4f40b81d1e0910257f83" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 37571 } } peers { permanent_uuid: "14ffbd8aab1646c395c9a689eb6a8ddc" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43877 } }
I20260928 06:49:06.332428 24935 leader_election.cc:288] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc [CANDIDATE]: Term 1 election: Requested vote from peers 28ecb9a2c8104db4a4751cbe5c094c35 (127.7.48.1:41747), 1a30b21ab77a4f40b81d1e0910257f83 (127.7.48.2:37571)
I20260928 06:49:06.332742 24599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7241cb56de4e36a5adacd9efa022c6" candidate_uuid: "14ffbd8aab1646c395c9a689eb6a8ddc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28ecb9a2c8104db4a4751cbe5c094c35"
I20260928 06:49:06.332880 24599 raft_consensus.cc:3037] T be7241cb56de4e36a5adacd9efa022c6 P 28ecb9a2c8104db4a4751cbe5c094c35 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:06.333047 24733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7241cb56de4e36a5adacd9efa022c6" candidate_uuid: "14ffbd8aab1646c395c9a689eb6a8ddc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a30b21ab77a4f40b81d1e0910257f83"
I20260928 06:49:06.333178 24733 raft_consensus.cc:3037] T be7241cb56de4e36a5adacd9efa022c6 P 1a30b21ab77a4f40b81d1e0910257f83 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:06.334316 24599 raft_consensus.cc:2454] T be7241cb56de4e36a5adacd9efa022c6 P 28ecb9a2c8104db4a4751cbe5c094c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ffbd8aab1646c395c9a689eb6a8ddc in term 1.
I20260928 06:49:06.334528 24733 raft_consensus.cc:2454] T be7241cb56de4e36a5adacd9efa022c6 P 1a30b21ab77a4f40b81d1e0910257f83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ffbd8aab1646c395c9a689eb6a8ddc in term 1.
I20260928 06:49:06.334703 24800 leader_election.cc:302] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ffbd8aab1646c395c9a689eb6a8ddc, 28ecb9a2c8104db4a4751cbe5c094c35; no voters:
I20260928 06:49:06.334892 24935 raft_consensus.cc:2781] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:06.335150 24935 raft_consensus.cc:686] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc [term 1 LEADER]: Becoming Leader. State: Replica: 14ffbd8aab1646c395c9a689eb6a8ddc, State: Running, Role: LEADER
I20260928 06:49:06.335487 24935 consensus_queue.cc:237] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ecb9a2c8104db4a4751cbe5c094c35" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 41747 } } peers { permanent_uuid: "1a30b21ab77a4f40b81d1e0910257f83" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 37571 } } peers { permanent_uuid: "14ffbd8aab1646c395c9a689eb6a8ddc" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43877 } }
I20260928 06:49:06.338933 24453 catalog_manager.cc:5495] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc reported cstate change: term changed from 0 to 1, leader changed from <none> to 14ffbd8aab1646c395c9a689eb6a8ddc (127.7.48.3). New cstate: current_term: 1 leader_uuid: "14ffbd8aab1646c395c9a689eb6a8ddc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ecb9a2c8104db4a4751cbe5c094c35" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 41747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a30b21ab77a4f40b81d1e0910257f83" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 37571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14ffbd8aab1646c395c9a689eb6a8ddc" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 43877 } health_report { overall_health: HEALTHY } } }
I20260928 06:49:06.355031 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:06.350714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33802 (local address 127.7.48.3:43877)
0928 06:49:06.350875 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:06.350880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:06.350896 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:49:06.351092 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:06.351095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:06.351154 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:06.351318 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:49:06.351588 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.352239 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.353302 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.353590 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.353736 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:06.354080 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:49:06.354534 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:49:06.354685 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:06.354756 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 06:49:06.354819 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 06:49:06.361368 24599 raft_consensus.cc:1261] T be7241cb56de4e36a5adacd9efa022c6 P 28ecb9a2c8104db4a4751cbe5c094c35 [term 1 FOLLOWER]: Refusing update from remote peer 14ffbd8aab1646c395c9a689eb6a8ddc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:49:06.361368 24733 raft_consensus.cc:1261] T be7241cb56de4e36a5adacd9efa022c6 P 1a30b21ab77a4f40b81d1e0910257f83 [term 1 FOLLOWER]: Refusing update from remote peer 14ffbd8aab1646c395c9a689eb6a8ddc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:49:06.362140 24935 consensus_queue.cc:1035] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc [LEADER]: Connected to new peer: Peer: permanent_uuid: "28ecb9a2c8104db4a4751cbe5c094c35" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 41747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:49:06.362362 24940 consensus_queue.cc:1035] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a30b21ab77a4f40b81d1e0910257f83" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 37571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:49:06.370955 24944 mvcc.cc:205] Tried to move back new op lower bound from 7334208087488352256 to 7334208087393378304. Current Snapshot: MvccSnapshot[applied={T|T < 7334208087488352256}]
I20260928 06:49:06.372843 24945 mvcc.cc:205] Tried to move back new op lower bound from 7334208087488352256 to 7334208087393378304. Current Snapshot: MvccSnapshot[applied={T|T < 7334208087488352256}]
I20260928 06:49:06.378476 24451 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:49:06.378688 24451 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:49:06.381399 24451 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 743e5d900bee4de7808f1510c3850217: Sending DeleteTablet for 3 replicas of tablet be7241cb56de4e36a5adacd9efa022c6
I20260928 06:49:06.382254 24713 tablet_service.cc:1514] Processing DeleteTablet for tablet be7241cb56de4e36a5adacd9efa022c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:49:06 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:41832
I20260928 06:49:06.382251 24579 tablet_service.cc:1514] Processing DeleteTablet for tablet be7241cb56de4e36a5adacd9efa022c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:49:06 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:56626
I20260928 06:49:06.382417 24847 tablet_service.cc:1514] Processing DeleteTablet for tablet be7241cb56de4e36a5adacd9efa022c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:49:06 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:33798
I20260928 06:49:06.382925 24954 tablet_replica.cc:330] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc: stopping tablet replica
I20260928 06:49:06.383025 24952 tablet_replica.cc:330] T be7241cb56de4e36a5adacd9efa022c6 P 28ecb9a2c8104db4a4751cbe5c094c35: stopping tablet replica
I20260928 06:49:06.383358 24952 raft_consensus.cc:2229] T be7241cb56de4e36a5adacd9efa022c6 P 28ecb9a2c8104db4a4751cbe5c094c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:49:06.383359 24954 raft_consensus.cc:2229] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:49:06.383620 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 24516
I20260928 06:49:06.385171 24953 tablet_replica.cc:330] T be7241cb56de4e36a5adacd9efa022c6 P 1a30b21ab77a4f40b81d1e0910257f83: stopping tablet replica
I20260928 06:49:06.385360 24954 raft_consensus.cc:2258] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:49:06.385494 24953 raft_consensus.cc:2229] T be7241cb56de4e36a5adacd9efa022c6 P 1a30b21ab77a4f40b81d1e0910257f83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:49:06.385916 24953 raft_consensus.cc:2258] T be7241cb56de4e36a5adacd9efa022c6 P 1a30b21ab77a4f40b81d1e0910257f83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:49:06.387280 24954 ts_tablet_manager.cc:1905] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:49:06.387593 24953 ts_tablet_manager.cc:1905] T be7241cb56de4e36a5adacd9efa022c6 P 1a30b21ab77a4f40b81d1e0910257f83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:49:06.390386 24953 ts_tablet_manager.cc:1918] T be7241cb56de4e36a5adacd9efa022c6 P 1a30b21ab77a4f40b81d1e0910257f83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:49:06.390491 24953 log.cc:1196] T be7241cb56de4e36a5adacd9efa022c6 P 1a30b21ab77a4f40b81d1e0910257f83: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/be7241cb56de4e36a5adacd9efa022c6
I20260928 06:49:06.390828 24953 ts_tablet_manager.cc:1939] T be7241cb56de4e36a5adacd9efa022c6 P 1a30b21ab77a4f40b81d1e0910257f83: Deleting consensus metadata
I20260928 06:49:06.391211 24954 ts_tablet_manager.cc:1918] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:49:06.391381 24954 log.cc:1196] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/be7241cb56de4e36a5adacd9efa022c6
I20260928 06:49:06.391785 24954 ts_tablet_manager.cc:1939] T be7241cb56de4e36a5adacd9efa022c6 P 14ffbd8aab1646c395c9a689eb6a8ddc: Deleting consensus metadata
I20260928 06:49:06.392185 24438 catalog_manager.cc:4841] TS 1a30b21ab77a4f40b81d1e0910257f83 (127.7.48.2:37571): tablet be7241cb56de4e36a5adacd9efa022c6 (table test-table [id=9d3ad394d17c4bc7beaa300065efdf48]) successfully deleted
W20260928 06:49:06.392428 24437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.48.1:41747 (error 108)
I20260928 06:49:06.393026 24439 catalog_manager.cc:4841] TS 14ffbd8aab1646c395c9a689eb6a8ddc (127.7.48.3:43877): tablet be7241cb56de4e36a5adacd9efa022c6 (table test-table [id=9d3ad394d17c4bc7beaa300065efdf48]) successfully deleted
I20260928 06:49:06.393296 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 24650
I20260928 06:49:06.393586 24925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:06.393197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:41747 (local address 127.0.0.1:56632)
0928 06:49:06.393411 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:06.393499 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.1:41747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260928 06:49:06.393989 24437 catalog_manager.cc:4584] TS 28ecb9a2c8104db4a4751cbe5c094c35 (127.7.48.1:41747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet be7241cb56de4e36a5adacd9efa022c6: Network error: Client connection negotiation failed: client connection to 127.7.48.1:41747: connect: Connection refused (error 111)
I20260928 06:49:06.400638 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 24784
I20260928 06:49:06.406584 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 24421
2026-09-28T06:49:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2287 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T06:49:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:06Z Disabled control of system clock
I20260928 06:49:06.437562 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:36507
--webserver_interface=127.7.48.62
--webserver_port=0
--builtin_ntp_servers=127.7.48.20:37577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:36507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:49:06.551646 24959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:06.551976 24959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:06.552014 24959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:06.555995 24959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:06.556057 24959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:06.556078 24959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:06.556097 24959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:06.556113 24959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:06.560900 24959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:37577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:36507
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:36507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:06.562057 24959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:06.563459 24959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:06.571955 24967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:06.571954 24964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:06.572183 24959 server_base.cc:1034] running on GCE node
W20260928 06:49:06.572043 24965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:06.572962 24959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:06.574159 24959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:06.575350 24959 hybrid_clock.cc:648] HybridClock initialized: now 1790578146575333 us; error 33 us; skew 500 ppm
I20260928 06:49:06.577342 24959 webserver.cc:460] Webserver started at http://127.7.48.62:44987/ using document root <none> and password file <none>
I20260928 06:49:06.577927 24959 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:06.578023 24959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:06.578265 24959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:06.580461 24959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "a7b8dda0f8944b5393e0a58d0556d5e8"
format_stamp: "Formatted at 2026-09-28 06:49:06 on dist-test-slave-twl6"
server_key: "57095270c4c7f74050034e62e9376b29"
server_key_iv: "2711e427fe9ea0fe144434a54cfd9c03"
server_key_version: "encryptionkey@0"
I20260928 06:49:06.581056 24959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "a7b8dda0f8944b5393e0a58d0556d5e8"
format_stamp: "Formatted at 2026-09-28 06:49:06 on dist-test-slave-twl6"
server_key: "57095270c4c7f74050034e62e9376b29"
server_key_iv: "2711e427fe9ea0fe144434a54cfd9c03"
server_key_version: "encryptionkey@0"
I20260928 06:49:06.584875 24959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 06:49:06.587430 24973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:06.588714 24959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 06:49:06.588888 24959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "a7b8dda0f8944b5393e0a58d0556d5e8"
format_stamp: "Formatted at 2026-09-28 06:49:06 on dist-test-slave-twl6"
server_key: "57095270c4c7f74050034e62e9376b29"
server_key_iv: "2711e427fe9ea0fe144434a54cfd9c03"
server_key_version: "encryptionkey@0"
I20260928 06:49:06.589007 24959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:06.613886 24959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:06.614686 24959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:06.614904 24959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:06.622058 24959 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:36507
I20260928 06:49:06.622085 25025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:36507 every 8 connection(s)
I20260928 06:49:06.623065 24959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:06.623850 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 24959
I20260928 06:49:06.624013 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:06.624337 7360 external_mini_cluster.cc:1428] Setting key 7d23785aeeeddd6a7a296448c31d4103
I20260928 06:49:06.626969 25026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:06.633131 25026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7b8dda0f8944b5393e0a58d0556d5e8: Bootstrap starting.
I20260928 06:49:06.634048 25029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:06.625820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38932 (local address 127.7.48.62:36507)
0928 06:49:06.626502 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:06.626518 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:06.626555 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:49:06.627289 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:06.627308 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:06.627721 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:06.628136 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:06.628595 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.629595 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.630527 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.631014 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.631201 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:06.632203 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:06.632208 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:06.632221 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:06.632626 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:06.632684 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:06.632704 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:06.632819 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:06.632970 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 06:49:06.633276 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":150,"threads_started":1}
I20260928 06:49:06.635708 25026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7b8dda0f8944b5393e0a58d0556d5e8: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:06.636655 25026 log.cc:824] T 00000000000000000000000000000000 P a7b8dda0f8944b5393e0a58d0556d5e8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:06.638523 25026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7b8dda0f8944b5393e0a58d0556d5e8: No bootstrap required, opened a new log
I20260928 06:49:06.642211 25026 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7b8dda0f8944b5393e0a58d0556d5e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b8dda0f8944b5393e0a58d0556d5e8" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36507 } }
I20260928 06:49:06.642519 25026 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7b8dda0f8944b5393e0a58d0556d5e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:06.642593 25026 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7b8dda0f8944b5393e0a58d0556d5e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7b8dda0f8944b5393e0a58d0556d5e8, State: Initialized, Role: FOLLOWER
I20260928 06:49:06.643080 25026 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7b8dda0f8944b5393e0a58d0556d5e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b8dda0f8944b5393e0a58d0556d5e8" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36507 } }
I20260928 06:49:06.643242 25026 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7b8dda0f8944b5393e0a58d0556d5e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:49:06.643332 25026 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7b8dda0f8944b5393e0a58d0556d5e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:49:06.643457 25026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7b8dda0f8944b5393e0a58d0556d5e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:06.644539 25026 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7b8dda0f8944b5393e0a58d0556d5e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b8dda0f8944b5393e0a58d0556d5e8" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36507 } }
I20260928 06:49:06.644894 25026 leader_election.cc:302] T 00000000000000000000000000000000 P a7b8dda0f8944b5393e0a58d0556d5e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7b8dda0f8944b5393e0a58d0556d5e8; no voters:
I20260928 06:49:06.645236 25026 leader_election.cc:288] T 00000000000000000000000000000000 P a7b8dda0f8944b5393e0a58d0556d5e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:49:06.645383 25031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7b8dda0f8944b5393e0a58d0556d5e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:06.645664 25031 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7b8dda0f8944b5393e0a58d0556d5e8 [term 1 LEADER]: Becoming Leader. State: Replica: a7b8dda0f8944b5393e0a58d0556d5e8, State: Running, Role: LEADER
I20260928 06:49:06.646000 25031 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7b8dda0f8944b5393e0a58d0556d5e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b8dda0f8944b5393e0a58d0556d5e8" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36507 } }
I20260928 06:49:06.646600 25026 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7b8dda0f8944b5393e0a58d0556d5e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:06.647792 25033 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7b8dda0f8944b5393e0a58d0556d5e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7b8dda0f8944b5393e0a58d0556d5e8. Latest consensus state: current_term: 1 leader_uuid: "a7b8dda0f8944b5393e0a58d0556d5e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b8dda0f8944b5393e0a58d0556d5e8" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36507 } } }
I20260928 06:49:06.647792 25032 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7b8dda0f8944b5393e0a58d0556d5e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7b8dda0f8944b5393e0a58d0556d5e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b8dda0f8944b5393e0a58d0556d5e8" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 36507 } } }
I20260928 06:49:06.648102 25032 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7b8dda0f8944b5393e0a58d0556d5e8 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:06.648138 25033 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7b8dda0f8944b5393e0a58d0556d5e8 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:06.649051 25040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:06.652281 25040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:49:06.657920 25040 catalog_manager.cc:1347] Generated new cluster ID: 014b7e651c3c48b389e6a5d0f1cd8c6e
I20260928 06:49:06.658028 25040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:06.679834 25040 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:06.680873 25040 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:06.687676 25040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7b8dda0f8944b5393e0a58d0556d5e8: Generated new TSK 0
I20260928 06:49:06.688488 25040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:49:06.696621 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:36507
--builtin_ntp_servers=127.7.48.20:37577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:49:06.809592 25050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:06.809860 25050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:06.809899 25050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:06.813941 25050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:06.813994 25050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:06.814049 25050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:49:06.818642 25050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:37577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:36507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:06.819715 25050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:06.821092 25050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:06.830900 25056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:06.830920 25058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:06.830892 25055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:06.831089 25050 server_base.cc:1034] running on GCE node
I20260928 06:49:06.831666 25050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:06.832387 25050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:06.833568 25050 hybrid_clock.cc:648] HybridClock initialized: now 1790578146833560 us; error 30 us; skew 500 ppm
I20260928 06:49:06.835400 25050 webserver.cc:460] Webserver started at http://127.7.48.1:37755/ using document root <none> and password file <none>
I20260928 06:49:06.836027 25050 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:06.836097 25050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:06.836269 25050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:06.838303 25050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "0c1b1d2df4184583b16f6d17c4852b7c"
format_stamp: "Formatted at 2026-09-28 06:49:06 on dist-test-slave-twl6"
server_key: "89098fde67bf49158a4feed98589690d"
server_key_iv: "2cc87187a7cc7075f8ce6320e3df8898"
server_key_version: "encryptionkey@0"
I20260928 06:49:06.838827 25050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "0c1b1d2df4184583b16f6d17c4852b7c"
format_stamp: "Formatted at 2026-09-28 06:49:06 on dist-test-slave-twl6"
server_key: "89098fde67bf49158a4feed98589690d"
server_key_iv: "2cc87187a7cc7075f8ce6320e3df8898"
server_key_version: "encryptionkey@0"
I20260928 06:49:06.842489 25050 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 06:49:06.844966 25064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:06.846288 25050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:49:06.846414 25050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "0c1b1d2df4184583b16f6d17c4852b7c"
format_stamp: "Formatted at 2026-09-28 06:49:06 on dist-test-slave-twl6"
server_key: "89098fde67bf49158a4feed98589690d"
server_key_iv: "2cc87187a7cc7075f8ce6320e3df8898"
server_key_version: "encryptionkey@0"
I20260928 06:49:06.846508 25050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:06.867257 25050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:06.867983 25050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:06.868137 25050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:06.868693 25050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:49:06.869598 25050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:49:06.869660 25050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:06.869701 25050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:49:06.869722 25050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:06.878268 25050 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:39297
I20260928 06:49:06.878291 25177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:39297 every 8 connection(s)
I20260928 06:49:06.879172 25050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:49:06.883323 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 25050
I20260928 06:49:06.883415 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
I20260928 06:49:06.883699 7360 external_mini_cluster.cc:1428] Setting key a323a5f44d95633fa065c4f3afa34327
I20260928 06:49:06.885869 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:36507
--builtin_ntp_servers=127.7.48.20:37577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:49:06.889324 25029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:06.880994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:33817 (local address 127.7.48.62:36507)
0928 06:49:06.881177 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:06.881182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:06.882241 (+ 1059us) server_negotiation.cc:407] Connection header received
0928 06:49:06.883647 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:06.883651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:06.883719 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:06.883813 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:06.884935 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.885701 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.886960 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.887371 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.887487 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:06.888731 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:06.888738 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:06.888744 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:06.888761 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:06.888788 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:06.888801 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:06.888882 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:06.889030 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 06:49:06.889110 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 06:49:06.890110 25180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:06.881411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:36507 (local address 127.7.48.1:33817)
0928 06:49:06.882086 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:06.882125 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:06.883394 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:06.884010 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:06.884016 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:06.884138 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:06.884796 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:06.884814 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.885834 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.885837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:06.886831 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:06.886841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:06.887541 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:06.887544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:06.887737 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:06.888453 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:49:06.888454 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:06.888496 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:49:06.888560 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:06.888890 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:06.888896 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:06.888900 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:49:06.888994 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 06:49:06.889283 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":127,"threads_started":1}
I20260928 06:49:06.891314 25178 heartbeater.cc:347] Connected to a master server at 127.7.48.62:36507
I20260928 06:49:06.891566 25178 heartbeater.cc:464] Registering TS with master...
I20260928 06:49:06.892159 25178 heartbeater.cc:511] Master 127.7.48.62:36507 requested a full tablet report, sending...
I20260928 06:49:06.893821 24990 ts_manager.cc:194] Registered new tserver with Master: 0c1b1d2df4184583b16f6d17c4852b7c (127.7.48.1:39297)
I20260928 06:49:06.894953 24990 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.48.1:33817
W20260928 06:49:06.997701 25181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:06.998018 25181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:06.998112 25181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:07.002218 25181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:07.002311 25181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:07.002405 25181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:49:07.007246 25181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:37577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:36507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:07.008471 25181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:07.009948 25181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:07.019719 25189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:07.019775 25186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:07.019719 25187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:07.020161 25181 server_base.cc:1034] running on GCE node
I20260928 06:49:07.020632 25181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:07.021241 25181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:07.022405 25181 hybrid_clock.cc:648] HybridClock initialized: now 1790578147022391 us; error 30 us; skew 500 ppm
I20260928 06:49:07.024297 25181 webserver.cc:460] Webserver started at http://127.7.48.2:41169/ using document root <none> and password file <none>
I20260928 06:49:07.024857 25181 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:07.024919 25181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:07.025089 25181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:07.027123 25181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "56b286fb800f46489fc9b0fe1e558ec3"
format_stamp: "Formatted at 2026-09-28 06:49:07 on dist-test-slave-twl6"
server_key: "ca08119c453c6af7b5e71f0c270b3687"
server_key_iv: "60b13ade84aa66ea2d353a46176f241d"
server_key_version: "encryptionkey@0"
I20260928 06:49:07.027639 25181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "56b286fb800f46489fc9b0fe1e558ec3"
format_stamp: "Formatted at 2026-09-28 06:49:07 on dist-test-slave-twl6"
server_key: "ca08119c453c6af7b5e71f0c270b3687"
server_key_iv: "60b13ade84aa66ea2d353a46176f241d"
server_key_version: "encryptionkey@0"
I20260928 06:49:07.031343 25181 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 06:49:07.033864 25195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:07.035100 25181 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:49:07.035229 25181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "56b286fb800f46489fc9b0fe1e558ec3"
format_stamp: "Formatted at 2026-09-28 06:49:07 on dist-test-slave-twl6"
server_key: "ca08119c453c6af7b5e71f0c270b3687"
server_key_iv: "60b13ade84aa66ea2d353a46176f241d"
server_key_version: "encryptionkey@0"
I20260928 06:49:07.035344 25181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:07.056872 25181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:07.057633 25181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:07.057835 25181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:07.058478 25181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:49:07.059464 25181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:49:07.059511 25181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:07.059587 25181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:49:07.059623 25181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:07.068948 25181 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:34929
I20260928 06:49:07.068967 25308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:34929 every 8 connection(s)
I20260928 06:49:07.069902 25181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:49:07.073195 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 25181
I20260928 06:49:07.073303 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:49:07.073624 7360 external_mini_cluster.cc:1428] Setting key e0223bb66f1640dd9fcd35260d211cad
I20260928 06:49:07.076058 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:36507
--builtin_ntp_servers=127.7.48.20:37577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:49:07.081115 25029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:07.071791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:60697 (local address 127.7.48.62:36507)
0928 06:49:07.071992 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:07.071995 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:07.073081 (+ 1086us) server_negotiation.cc:407] Connection header received
0928 06:49:07.074933 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:07.074935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:07.075003 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:07.075089 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:07.076499 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.077282 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.079014 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.079261 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.079381 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:07.080586 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:07.080591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:07.080599 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:07.080616 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:07.080634 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:07.080642 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:07.080720 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:07.080899 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 06:49:07.080968 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 06:49:07.082037 25311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:07.072257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:36507 (local address 127.7.48.2:60697)
0928 06:49:07.072921 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:07.072965 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:07.074704 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:07.075343 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:07.075355 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:07.075475 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:07.076309 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:07.076332 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.077426 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.077428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:07.078879 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:07.078895 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.079429 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.079433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:07.079620 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:07.080316 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:49:07.080318 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:07.080359 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:49:07.080430 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:07.080734 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:07.080744 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:07.080750 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:49:07.080885 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 06:49:07.081194 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":118,"threads_started":1}
I20260928 06:49:07.083416 25309 heartbeater.cc:347] Connected to a master server at 127.7.48.62:36507
I20260928 06:49:07.083681 25309 heartbeater.cc:464] Registering TS with master...
I20260928 06:49:07.084314 25309 heartbeater.cc:511] Master 127.7.48.62:36507 requested a full tablet report, sending...
I20260928 06:49:07.085495 24990 ts_manager.cc:194] Registered new tserver with Master: 56b286fb800f46489fc9b0fe1e558ec3 (127.7.48.2:34929)
I20260928 06:49:07.086423 24990 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.48.2:60697
W20260928 06:49:07.191057 25312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:07.191335 25312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:07.191375 25312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:07.195761 25312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:07.195835 25312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:07.195972 25312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:49:07.200806 25312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:37577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:36507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:07.201876 25312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:07.203250 25312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:07.213455 25321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:07.213456 25318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:07.213486 25319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:07.213518 25312 server_base.cc:1034] running on GCE node
I20260928 06:49:07.214226 25312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:07.214893 25312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:07.216059 25312 hybrid_clock.cc:648] HybridClock initialized: now 1790578147216042 us; error 27 us; skew 500 ppm
I20260928 06:49:07.217890 25312 webserver.cc:460] Webserver started at http://127.7.48.3:46871/ using document root <none> and password file <none>
I20260928 06:49:07.218652 25312 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:07.218761 25312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:07.219077 25312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:07.221686 25312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "b453be29161d44cab63edadc1358b1e3"
format_stamp: "Formatted at 2026-09-28 06:49:07 on dist-test-slave-twl6"
server_key: "a7cf46e1107e444f1d2e93eae244c080"
server_key_iv: "edf67f5ee19f7db9adbd6ab3656223fd"
server_key_version: "encryptionkey@0"
I20260928 06:49:07.222347 25312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "b453be29161d44cab63edadc1358b1e3"
format_stamp: "Formatted at 2026-09-28 06:49:07 on dist-test-slave-twl6"
server_key: "a7cf46e1107e444f1d2e93eae244c080"
server_key_iv: "edf67f5ee19f7db9adbd6ab3656223fd"
server_key_version: "encryptionkey@0"
I20260928 06:49:07.226468 25312 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:49:07.229183 25327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:07.230414 25312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:49:07.230546 25312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "b453be29161d44cab63edadc1358b1e3"
format_stamp: "Formatted at 2026-09-28 06:49:07 on dist-test-slave-twl6"
server_key: "a7cf46e1107e444f1d2e93eae244c080"
server_key_iv: "edf67f5ee19f7db9adbd6ab3656223fd"
server_key_version: "encryptionkey@0"
I20260928 06:49:07.230659 25312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:07.245426 25312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:07.246171 25312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:07.246397 25312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:07.247159 25312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:49:07.248342 25312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:49:07.248425 25312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:07.248492 25312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:49:07.248540 25312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:07.257761 25312 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:42757
I20260928 06:49:07.257827 25440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:42757 every 8 connection(s)
I20260928 06:49:07.258783 25312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:49:07.263033 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 25312
I20260928 06:49:07.263159 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:49:07.263489 7360 external_mini_cluster.cc:1428] Setting key 8de56ccb3a546e653704b9c0c86eeaaa
I20260928 06:49:07.269124 25029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:07.260750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:58187 (local address 127.7.48.62:36507)
0928 06:49:07.260910 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:07.260915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:07.262076 (+ 1161us) server_negotiation.cc:407] Connection header received
0928 06:49:07.263733 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:07.263735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:07.263798 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:07.263955 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:07.265183 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.265853 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.267054 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.267277 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.267393 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:07.268597 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:07.268604 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:07.268607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:07.268619 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:07.268640 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:07.268649 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:07.268729 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:07.268909 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 06:49:07.268975 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 06:49:07.270025 25443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:07.261226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:36507 (local address 127.7.48.3:58187)
0928 06:49:07.261892 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:07.261932 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:07.263490 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:07.264153 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:07.264162 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:07.264296 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:07.265039 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:07.265055 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.265986 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.265990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:07.266926 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:07.266935 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.267439 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.267444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:07.267630 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:07.268306 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:49:07.268308 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:07.268355 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:49:07.268425 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:07.268744 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:07.268753 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:07.268758 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:49:07.268898 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 06:49:07.269217 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":125,"threads_started":1}
I20260928 06:49:07.271132 25441 heartbeater.cc:347] Connected to a master server at 127.7.48.62:36507
I20260928 06:49:07.271384 25441 heartbeater.cc:464] Registering TS with master...
I20260928 06:49:07.271979 25441 heartbeater.cc:511] Master 127.7.48.62:36507 requested a full tablet report, sending...
I20260928 06:49:07.273027 24990 ts_manager.cc:194] Registered new tserver with Master: b453be29161d44cab63edadc1358b1e3 (127.7.48.3:42757)
I20260928 06:49:07.273630 24990 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.48.3:58187
I20260928 06:49:07.277457 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:49:07.282860 25029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:07.279433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38936 (local address 127.7.48.62:36507)
0928 06:49:07.279561 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:07.279565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:07.279769 (+ 204us) server_negotiation.cc:407] Connection header received
0928 06:49:07.279868 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:07.279870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:07.279953 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:07.280012 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:07.280232 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.280707 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.281453 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.281653 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.281722 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:07.282531 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:07.282533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:07.282536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:07.282546 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:07.282560 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:07.282567 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:07.282618 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:07.282704 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 06:49:07.282750 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 06:49:07.285902 24990 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:49:07.289039 24990 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:49:07.303684 25453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:07.298021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39038 (local address 127.7.48.1:39297)
0928 06:49:07.298825 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:07.298837 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:07.298862 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:49:07.298943 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:07.298951 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:07.299191 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:07.299750 (+ 559us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:07.299933 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.300821 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.302172 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.303005 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.303134 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:07.303349 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:07.303408 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 06:49:07.303503 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":135,"threads_started":1}
I20260928 06:49:07.303678 25452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:07.297877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:39297 (local address 127.0.0.1:39038)
0928 06:49:07.298303 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:07.298375 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:07.298515 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:07.299383 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:07.299390 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:07.299416 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:07.299584 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:07.299591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.300937 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.300942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:07.302026 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:07.302035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.303128 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.303132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:07.303237 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:07.303274 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 06:49:07.303381 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 06:49:07.303502 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":103,"threads_started":1}
I20260928 06:49:07.304656 25454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:07.298275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:34929 (local address 127.0.0.1:41434)
0928 06:49:07.299205 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:07.299221 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:07.299305 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:07.299943 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:07.299945 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:07.299960 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:07.300086 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:07.300093 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.301555 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.301560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:07.302783 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:07.302796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.304269 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.304272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:07.304366 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:07.304390 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 06:49:07.304456 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:49:07.304517 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":28,"thread_start_us":121,"threads_started":1}
I20260928 06:49:07.304893 25456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:07.298780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41434 (local address 127.7.48.2:34929)
0928 06:49:07.299454 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:07.299464 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:07.299486 (+ 22us) server_negotiation.cc:407] Connection header received
0928 06:49:07.299586 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:07.299591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:07.299792 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:07.299975 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:07.300231 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.301407 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.302930 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.303727 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.304268 (+ 541us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:07.304417 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:07.304481 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 06:49:07.304587 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":94,"threads_started":1}
I20260928 06:49:07.305867 25455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:07.298722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:42757 (local address 127.0.0.1:60128)
0928 06:49:07.299495 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:07.299504 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:07.299628 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:07.300131 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:07.300134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:07.300152 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:07.300230 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:07.300235 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.301593 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.301596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:07.303187 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:07.303198 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.304478 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.304482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:07.304566 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:07.304589 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 06:49:07.304662 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:49:07.305746 (+ 1084us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":84,"thread_start_us":99,"threads_started":1}
I20260928 06:49:07.306284 25457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:07.298889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60128 (local address 127.7.48.3:42757)
0928 06:49:07.299266 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:07.299280 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:07.299626 (+ 346us) server_negotiation.cc:407] Connection header received
0928 06:49:07.299747 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:07.299754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:07.299960 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:07.300105 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:07.300346 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.301475 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.303331 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.304117 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.304238 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:07.304413 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:07.306031 (+ 1618us) server_negotiation.cc:299] Negotiation successful
0928 06:49:07.306131 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":92,"threads_started":1}
I20260928 06:49:07.307098 25112 tablet_service.cc:1467] Processing CreateTablet for tablet 1abc1bb793ae4fe9a74d8744898e6060 (DEFAULT_TABLE table=test-table [id=c6e6b27822354fb0b57448ceab152ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:49:07.307482 25243 tablet_service.cc:1467] Processing CreateTablet for tablet 1abc1bb793ae4fe9a74d8744898e6060 (DEFAULT_TABLE table=test-table [id=c6e6b27822354fb0b57448ceab152ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:49:07.308357 25112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1abc1bb793ae4fe9a74d8744898e6060. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:07.308490 25243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1abc1bb793ae4fe9a74d8744898e6060. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:07.309062 25375 tablet_service.cc:1467] Processing CreateTablet for tablet 1abc1bb793ae4fe9a74d8744898e6060 (DEFAULT_TABLE table=test-table [id=c6e6b27822354fb0b57448ceab152ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:49:07.310042 25375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1abc1bb793ae4fe9a74d8744898e6060. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:07.315659 25458 tablet_bootstrap.cc:492] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3: Bootstrap starting.
I20260928 06:49:07.315981 25459 tablet_bootstrap.cc:492] T 1abc1bb793ae4fe9a74d8744898e6060 P 0c1b1d2df4184583b16f6d17c4852b7c: Bootstrap starting.
I20260928 06:49:07.316915 25460 tablet_bootstrap.cc:492] T 1abc1bb793ae4fe9a74d8744898e6060 P b453be29161d44cab63edadc1358b1e3: Bootstrap starting.
I20260928 06:49:07.318099 25459 tablet_bootstrap.cc:654] T 1abc1bb793ae4fe9a74d8744898e6060 P 0c1b1d2df4184583b16f6d17c4852b7c: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:07.318202 25458 tablet_bootstrap.cc:654] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:07.318884 25459 log.cc:824] T 1abc1bb793ae4fe9a74d8744898e6060 P 0c1b1d2df4184583b16f6d17c4852b7c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:07.318997 25460 tablet_bootstrap.cc:654] T 1abc1bb793ae4fe9a74d8744898e6060 P b453be29161d44cab63edadc1358b1e3: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:07.319043 25458 log.cc:824] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:07.319739 25460 log.cc:824] T 1abc1bb793ae4fe9a74d8744898e6060 P b453be29161d44cab63edadc1358b1e3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:07.321116 25458 tablet_bootstrap.cc:492] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3: No bootstrap required, opened a new log
I20260928 06:49:07.321174 25459 tablet_bootstrap.cc:492] T 1abc1bb793ae4fe9a74d8744898e6060 P 0c1b1d2df4184583b16f6d17c4852b7c: No bootstrap required, opened a new log
I20260928 06:49:07.321348 25458 ts_tablet_manager.cc:1397] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 06:49:07.321390 25459 ts_tablet_manager.cc:1397] T 1abc1bb793ae4fe9a74d8744898e6060 P 0c1b1d2df4184583b16f6d17c4852b7c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 06:49:07.322679 25460 tablet_bootstrap.cc:492] T 1abc1bb793ae4fe9a74d8744898e6060 P b453be29161d44cab63edadc1358b1e3: No bootstrap required, opened a new log
I20260928 06:49:07.322862 25460 ts_tablet_manager.cc:1397] T 1abc1bb793ae4fe9a74d8744898e6060 P b453be29161d44cab63edadc1358b1e3: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260928 06:49:07.324787 25458 raft_consensus.cc:348] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1b1d2df4184583b16f6d17c4852b7c" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 39297 } } peers { permanent_uuid: "b453be29161d44cab63edadc1358b1e3" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 42757 } } peers { permanent_uuid: "56b286fb800f46489fc9b0fe1e558ec3" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 34929 } }
I20260928 06:49:07.325033 25458 raft_consensus.cc:374] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:07.325095 25458 raft_consensus.cc:729] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56b286fb800f46489fc9b0fe1e558ec3, State: Initialized, Role: FOLLOWER
I20260928 06:49:07.325556 25458 consensus_queue.cc:260] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1b1d2df4184583b16f6d17c4852b7c" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 39297 } } peers { permanent_uuid: "b453be29161d44cab63edadc1358b1e3" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 42757 } } peers { permanent_uuid: "56b286fb800f46489fc9b0fe1e558ec3" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 34929 } }
I20260928 06:49:07.325722 25459 raft_consensus.cc:348] T 1abc1bb793ae4fe9a74d8744898e6060 P 0c1b1d2df4184583b16f6d17c4852b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1b1d2df4184583b16f6d17c4852b7c" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 39297 } } peers { permanent_uuid: "b453be29161d44cab63edadc1358b1e3" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 42757 } } peers { permanent_uuid: "56b286fb800f46489fc9b0fe1e558ec3" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 34929 } }
I20260928 06:49:07.325981 25459 raft_consensus.cc:374] T 1abc1bb793ae4fe9a74d8744898e6060 P 0c1b1d2df4184583b16f6d17c4852b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:07.326040 25459 raft_consensus.cc:729] T 1abc1bb793ae4fe9a74d8744898e6060 P 0c1b1d2df4184583b16f6d17c4852b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1b1d2df4184583b16f6d17c4852b7c, State: Initialized, Role: FOLLOWER
I20260928 06:49:07.326370 25309 heartbeater.cc:503] Master 127.7.48.62:36507 was elected leader, sending a full tablet report...
I20260928 06:49:07.326464 25459 consensus_queue.cc:260] T 1abc1bb793ae4fe9a74d8744898e6060 P 0c1b1d2df4184583b16f6d17c4852b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1b1d2df4184583b16f6d17c4852b7c" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 39297 } } peers { permanent_uuid: "b453be29161d44cab63edadc1358b1e3" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 42757 } } peers { permanent_uuid: "56b286fb800f46489fc9b0fe1e558ec3" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 34929 } }
I20260928 06:49:07.326694 25458 ts_tablet_manager.cc:1428] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260928 06:49:07.327014 25460 raft_consensus.cc:348] T 1abc1bb793ae4fe9a74d8744898e6060 P b453be29161d44cab63edadc1358b1e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1b1d2df4184583b16f6d17c4852b7c" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 39297 } } peers { permanent_uuid: "b453be29161d44cab63edadc1358b1e3" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 42757 } } peers { permanent_uuid: "56b286fb800f46489fc9b0fe1e558ec3" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 34929 } }
I20260928 06:49:07.327272 25460 raft_consensus.cc:374] T 1abc1bb793ae4fe9a74d8744898e6060 P b453be29161d44cab63edadc1358b1e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:07.327329 25460 raft_consensus.cc:729] T 1abc1bb793ae4fe9a74d8744898e6060 P b453be29161d44cab63edadc1358b1e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b453be29161d44cab63edadc1358b1e3, State: Initialized, Role: FOLLOWER
I20260928 06:49:07.327442 25459 ts_tablet_manager.cc:1428] T 1abc1bb793ae4fe9a74d8744898e6060 P 0c1b1d2df4184583b16f6d17c4852b7c: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260928 06:49:07.327785 25178 heartbeater.cc:503] Master 127.7.48.62:36507 was elected leader, sending a full tablet report...
I20260928 06:49:07.327970 25460 consensus_queue.cc:260] T 1abc1bb793ae4fe9a74d8744898e6060 P b453be29161d44cab63edadc1358b1e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1b1d2df4184583b16f6d17c4852b7c" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 39297 } } peers { permanent_uuid: "b453be29161d44cab63edadc1358b1e3" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 42757 } } peers { permanent_uuid: "56b286fb800f46489fc9b0fe1e558ec3" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 34929 } }
I20260928 06:49:07.328918 25460 ts_tablet_manager.cc:1428] T 1abc1bb793ae4fe9a74d8744898e6060 P b453be29161d44cab63edadc1358b1e3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 06:49:07.329267 25441 heartbeater.cc:503] Master 127.7.48.62:36507 was elected leader, sending a full tablet report...
W20260928 06:49:07.380388 25179 tablet.cc:2369] T 1abc1bb793ae4fe9a74d8744898e6060 P 0c1b1d2df4184583b16f6d17c4852b7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:49:07.398062 25464 raft_consensus.cc:482] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:49:07.398275 25464 raft_consensus.cc:504] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1b1d2df4184583b16f6d17c4852b7c" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 39297 } } peers { permanent_uuid: "b453be29161d44cab63edadc1358b1e3" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 42757 } } peers { permanent_uuid: "56b286fb800f46489fc9b0fe1e558ec3" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 34929 } }
I20260928 06:49:07.399327 25464 leader_election.cc:288] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c1b1d2df4184583b16f6d17c4852b7c (127.7.48.1:39297), b453be29161d44cab63edadc1358b1e3 (127.7.48.3:42757)
I20260928 06:49:07.402915 25453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:07.399736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:56855 (local address 127.7.48.1:39297)
0928 06:49:07.399912 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:07.399917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:07.399936 (+ 19us) server_negotiation.cc:407] Connection header received
0928 06:49:07.400001 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:07.400006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:07.400066 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:07.400149 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:07.400464 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.400963 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.401993 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.402369 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.402485 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:07.402543 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:07.402727 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 06:49:07.402792 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 06:49:07.402915 25311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:07.399582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:39297 (local address 127.7.48.2:56855)
0928 06:49:07.399737 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:07.399754 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:07.399863 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:07.400195 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:07.400198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:07.400214 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:07.400345 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:07.400351 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.401089 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.401093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:07.401867 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:07.401875 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.402491 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.402494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:07.402594 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:07.402622 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 06:49:07.402708 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:49:07.402779 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 06:49:07.403512 25457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:07.400028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:58387 (local address 127.7.48.3:42757)
0928 06:49:07.400187 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:07.400192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:07.400343 (+ 151us) server_negotiation.cc:407] Connection header received
0928 06:49:07.400490 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:07.400493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:07.400541 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:07.400628 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:07.400898 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.401385 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.402398 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.402789 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.402907 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:07.402962 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:07.403335 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 06:49:07.403413 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 06:49:07.403523 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:07.399859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:42757 (local address 127.7.48.2:58387)
0928 06:49:07.400258 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:07.400269 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:07.400382 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:07.400649 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:07.400652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:07.400668 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:07.400792 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:07.400797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.401503 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.401507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:07.402264 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:07.402273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.402912 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.402916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:07.403166 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:07.403189 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 06:49:07.403279 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 06:49:07.403385 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":139,"threads_started":1}
I20260928 06:49:07.403707 25132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abc1bb793ae4fe9a74d8744898e6060" candidate_uuid: "56b286fb800f46489fc9b0fe1e558ec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c1b1d2df4184583b16f6d17c4852b7c" is_pre_election: true
I20260928 06:49:07.404062 25132 raft_consensus.cc:2454] T 1abc1bb793ae4fe9a74d8744898e6060 P 0c1b1d2df4184583b16f6d17c4852b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56b286fb800f46489fc9b0fe1e558ec3 in term 0.
I20260928 06:49:07.404165 25395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abc1bb793ae4fe9a74d8744898e6060" candidate_uuid: "56b286fb800f46489fc9b0fe1e558ec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b453be29161d44cab63edadc1358b1e3" is_pre_election: true
I20260928 06:49:07.404475 25395 raft_consensus.cc:2454] T 1abc1bb793ae4fe9a74d8744898e6060 P b453be29161d44cab63edadc1358b1e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56b286fb800f46489fc9b0fe1e558ec3 in term 0.
I20260928 06:49:07.404583 25197 leader_election.cc:302] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c1b1d2df4184583b16f6d17c4852b7c, 56b286fb800f46489fc9b0fe1e558ec3; no voters:
I20260928 06:49:07.404911 25464 raft_consensus.cc:2781] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:49:07.404999 25464 raft_consensus.cc:482] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:49:07.405042 25464 raft_consensus.cc:3037] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:07.406040 25464 raft_consensus.cc:504] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1b1d2df4184583b16f6d17c4852b7c" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 39297 } } peers { permanent_uuid: "b453be29161d44cab63edadc1358b1e3" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 42757 } } peers { permanent_uuid: "56b286fb800f46489fc9b0fe1e558ec3" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 34929 } }
I20260928 06:49:07.406368 25464 leader_election.cc:288] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3 [CANDIDATE]: Term 1 election: Requested vote from peers 0c1b1d2df4184583b16f6d17c4852b7c (127.7.48.1:39297), b453be29161d44cab63edadc1358b1e3 (127.7.48.3:42757)
I20260928 06:49:07.406683 25132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abc1bb793ae4fe9a74d8744898e6060" candidate_uuid: "56b286fb800f46489fc9b0fe1e558ec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c1b1d2df4184583b16f6d17c4852b7c"
I20260928 06:49:07.406749 25395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abc1bb793ae4fe9a74d8744898e6060" candidate_uuid: "56b286fb800f46489fc9b0fe1e558ec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b453be29161d44cab63edadc1358b1e3"
I20260928 06:49:07.406827 25132 raft_consensus.cc:3037] T 1abc1bb793ae4fe9a74d8744898e6060 P 0c1b1d2df4184583b16f6d17c4852b7c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:07.406905 25395 raft_consensus.cc:3037] T 1abc1bb793ae4fe9a74d8744898e6060 P b453be29161d44cab63edadc1358b1e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:07.407855 25132 raft_consensus.cc:2454] T 1abc1bb793ae4fe9a74d8744898e6060 P 0c1b1d2df4184583b16f6d17c4852b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56b286fb800f46489fc9b0fe1e558ec3 in term 1.
I20260928 06:49:07.407977 25395 raft_consensus.cc:2454] T 1abc1bb793ae4fe9a74d8744898e6060 P b453be29161d44cab63edadc1358b1e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56b286fb800f46489fc9b0fe1e558ec3 in term 1.
I20260928 06:49:07.408223 25197 leader_election.cc:302] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c1b1d2df4184583b16f6d17c4852b7c, 56b286fb800f46489fc9b0fe1e558ec3; no voters:
I20260928 06:49:07.408375 25464 raft_consensus.cc:2781] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:07.408581 25464 raft_consensus.cc:686] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3 [term 1 LEADER]: Becoming Leader. State: Replica: 56b286fb800f46489fc9b0fe1e558ec3, State: Running, Role: LEADER
I20260928 06:49:07.408919 25464 consensus_queue.cc:237] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1b1d2df4184583b16f6d17c4852b7c" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 39297 } } peers { permanent_uuid: "b453be29161d44cab63edadc1358b1e3" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 42757 } } peers { permanent_uuid: "56b286fb800f46489fc9b0fe1e558ec3" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 34929 } }
I20260928 06:49:07.412207 24990 catalog_manager.cc:5495] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56b286fb800f46489fc9b0fe1e558ec3 (127.7.48.2). New cstate: current_term: 1 leader_uuid: "56b286fb800f46489fc9b0fe1e558ec3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1b1d2df4184583b16f6d17c4852b7c" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 39297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b453be29161d44cab63edadc1358b1e3" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 42757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56b286fb800f46489fc9b0fe1e558ec3" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 34929 } health_report { overall_health: HEALTHY } } }
I20260928 06:49:07.427501 25456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:07.424238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.7.48.2:34929)
0928 06:49:07.424370 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:07.424374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:07.424388 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:49:07.424461 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:07.424463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:07.424517 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:07.424619 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:49:07.424814 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.425314 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.426104 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:07.426305 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:07.426398 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:07.426659 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:49:07.427084 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:49:07.427178 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:07.427236 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 06:49:07.427285 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:49:07.433409 25395 raft_consensus.cc:1261] T 1abc1bb793ae4fe9a74d8744898e6060 P b453be29161d44cab63edadc1358b1e3 [term 1 FOLLOWER]: Refusing update from remote peer 56b286fb800f46489fc9b0fe1e558ec3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:49:07.433419 25132 raft_consensus.cc:1261] T 1abc1bb793ae4fe9a74d8744898e6060 P 0c1b1d2df4184583b16f6d17c4852b7c [term 1 FOLLOWER]: Refusing update from remote peer 56b286fb800f46489fc9b0fe1e558ec3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:49:07.434216 25468 consensus_queue.cc:1035] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b453be29161d44cab63edadc1358b1e3" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 42757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:49:07.434445 25464 consensus_queue.cc:1035] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c1b1d2df4184583b16f6d17c4852b7c" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 39297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:49:07.443257 25473 mvcc.cc:205] Tried to move back new op lower bound from 7334208091879862272 to 7334208091789631488. Current Snapshot: MvccSnapshot[applied={T|T < 7334208091879862272}]
I20260928 06:49:07.452512 24988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:49:07.452683 24988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:49:07.455298 24988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7b8dda0f8944b5393e0a58d0556d5e8: Sending DeleteTablet for 3 replicas of tablet 1abc1bb793ae4fe9a74d8744898e6060
I20260928 06:49:07.456271 25243 tablet_service.cc:1514] Processing DeleteTablet for tablet 1abc1bb793ae4fe9a74d8744898e6060 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:49:07 UTC) from {username='slave'} at 127.0.0.1:41434
I20260928 06:49:07.456274 25112 tablet_service.cc:1514] Processing DeleteTablet for tablet 1abc1bb793ae4fe9a74d8744898e6060 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:49:07 UTC) from {username='slave'} at 127.0.0.1:39038
I20260928 06:49:07.456946 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 25050
I20260928 06:49:07.457057 25375 tablet_service.cc:1514] Processing DeleteTablet for tablet 1abc1bb793ae4fe9a74d8744898e6060 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:49:07 UTC) from {username='slave'} at 127.0.0.1:60128
I20260928 06:49:07.457077 25481 tablet_replica.cc:330] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3: stopping tablet replica
I20260928 06:49:07.457974 25481 raft_consensus.cc:2229] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:49:07.458442 25481 raft_consensus.cc:2258] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:49:07.459720 25483 tablet_replica.cc:330] T 1abc1bb793ae4fe9a74d8744898e6060 P b453be29161d44cab63edadc1358b1e3: stopping tablet replica
I20260928 06:49:07.460052 25481 ts_tablet_manager.cc:1905] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:49:07.460196 25483 raft_consensus.cc:2229] T 1abc1bb793ae4fe9a74d8744898e6060 P b453be29161d44cab63edadc1358b1e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:49:07.460611 25483 raft_consensus.cc:2258] T 1abc1bb793ae4fe9a74d8744898e6060 P b453be29161d44cab63edadc1358b1e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:49:07.462164 25483 ts_tablet_manager.cc:1905] T 1abc1bb793ae4fe9a74d8744898e6060 P b453be29161d44cab63edadc1358b1e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:49:07.463682 25481 ts_tablet_manager.cc:1918] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:49:07.463789 25481 log.cc:1196] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/1abc1bb793ae4fe9a74d8744898e6060
I20260928 06:49:07.464138 25481 ts_tablet_manager.cc:1939] T 1abc1bb793ae4fe9a74d8744898e6060 P 56b286fb800f46489fc9b0fe1e558ec3: Deleting consensus metadata
I20260928 06:49:07.465186 25483 ts_tablet_manager.cc:1918] T 1abc1bb793ae4fe9a74d8744898e6060 P b453be29161d44cab63edadc1358b1e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:49:07.465350 25483 log.cc:1196] T 1abc1bb793ae4fe9a74d8744898e6060 P b453be29161d44cab63edadc1358b1e3: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/1abc1bb793ae4fe9a74d8744898e6060
W20260928 06:49:07.465415 24975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.48.1:39297 (error 108)
I20260928 06:49:07.465685 25483 ts_tablet_manager.cc:1939] T 1abc1bb793ae4fe9a74d8744898e6060 P b453be29161d44cab63edadc1358b1e3: Deleting consensus metadata
I20260928 06:49:07.466090 24976 catalog_manager.cc:4841] TS 56b286fb800f46489fc9b0fe1e558ec3 (127.7.48.2:34929): tablet 1abc1bb793ae4fe9a74d8744898e6060 (table test-table [id=c6e6b27822354fb0b57448ceab152ca6]) successfully deleted
W20260928 06:49:07.466799 25455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:07.466457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:39297 (local address 127.0.0.1:39052)
0928 06:49:07.466595 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:07.466687 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.1:39297: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 06:49:07.466854 24975 catalog_manager.cc:4841] TS b453be29161d44cab63edadc1358b1e3 (127.7.48.3:42757): tablet 1abc1bb793ae4fe9a74d8744898e6060 (table test-table [id=c6e6b27822354fb0b57448ceab152ca6]) successfully deleted
I20260928 06:49:07.466989 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 25181
W20260928 06:49:07.467031 24975 catalog_manager.cc:4584] TS 0c1b1d2df4184583b16f6d17c4852b7c (127.7.48.1:39297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1abc1bb793ae4fe9a74d8744898e6060: Network error: Client connection negotiation failed: client connection to 127.7.48.1:39297: connect: Connection reset by peer (error 104)
I20260928 06:49:07.473362 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 25312
I20260928 06:49:07.478899 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 24959
2026-09-28T06:49:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1071 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:49:07 dist-test-slave-twl6 krb5kdc[25487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:49:07 dist-test-slave-twl6 krb5kdc[25487](info): set up 2 sockets
Sep 28 06:49:07 dist-test-slave-twl6 krb5kdc[25487](info): commencing operation
krb5kdc: starting...
W20260928 06:49:08.492095 7360 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:49:08 dist-test-slave-twl6 krb5kdc[25487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:49:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:49:08.615070 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:34121
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:33417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:34121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:08.725889 25500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:08.726238 25500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:08.726317 25500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:08.730391 25500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:08.730506 25500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:08.730543 25500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:08.730583 25500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:08.730621 25500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:08.736084 25500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:33417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:34121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:34121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:08.737252 25500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:08.738373 25500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:08.747099 25506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:08.747099 25505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:08.747092 25508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:08.747413 25500 server_base.cc:1034] running on GCE node
I20260928 06:49:08.748150 25500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:08.749228 25500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:08.750418 25500 hybrid_clock.cc:648] HybridClock initialized: now 1790578148750400 us; error 31 us; skew 500 ppm
Sep 28 06:49:08 dist-test-slave-twl6 krb5kdc[25487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578148, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:08.753211 25500 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:49:08.754173 25500 webserver.cc:460] Webserver started at http://127.7.48.62:36235/ using document root <none> and password file <none>
I20260928 06:49:08.754760 25500 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:08.754854 25500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:08.755108 25500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:08.757288 25500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "afee9a4cc7c74f85bcc21ae61fab2a2b"
format_stamp: "Formatted at 2026-09-28 06:49:08 on dist-test-slave-twl6"
server_key: "628adc2d0a983b81a6a469fe9eae0d42"
server_key_iv: "d1524a0084aafaca41dfcefaae2a96ea"
server_key_version: "encryptionkey@0"
I20260928 06:49:08.757884 25500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "afee9a4cc7c74f85bcc21ae61fab2a2b"
format_stamp: "Formatted at 2026-09-28 06:49:08 on dist-test-slave-twl6"
server_key: "628adc2d0a983b81a6a469fe9eae0d42"
server_key_iv: "d1524a0084aafaca41dfcefaae2a96ea"
server_key_version: "encryptionkey@0"
I20260928 06:49:08.761828 25500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:49:08.764380 25515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:08.765697 25500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:49:08.765846 25500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "afee9a4cc7c74f85bcc21ae61fab2a2b"
format_stamp: "Formatted at 2026-09-28 06:49:08 on dist-test-slave-twl6"
server_key: "628adc2d0a983b81a6a469fe9eae0d42"
server_key_iv: "d1524a0084aafaca41dfcefaae2a96ea"
server_key_version: "encryptionkey@0"
I20260928 06:49:08.765962 25500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:08.782437 25500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:08.785511 25500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:08.785732 25500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:08.793109 25500 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:34121
I20260928 06:49:08.793129 25567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:34121 every 8 connection(s)
I20260928 06:49:08.794102 25500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:08.796880 25568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:08.801146 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 25500
I20260928 06:49:08.801298 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:08.801594 7360 external_mini_cluster.cc:1428] Setting key 48a0f60720b211ab8c8e43d4b4842768
I20260928 06:49:08.802999 25568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afee9a4cc7c74f85bcc21ae61fab2a2b: Bootstrap starting.
I20260928 06:49:08.805322 25568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afee9a4cc7c74f85bcc21ae61fab2a2b: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:08.806131 25568 log.cc:824] T 00000000000000000000000000000000 P afee9a4cc7c74f85bcc21ae61fab2a2b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:08.808414 25568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afee9a4cc7c74f85bcc21ae61fab2a2b: No bootstrap required, opened a new log
Sep 28 06:49:08 dist-test-slave-twl6 krb5kdc[25487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:49:08.812665 25568 raft_consensus.cc:348] T 00000000000000000000000000000000 P afee9a4cc7c74f85bcc21ae61fab2a2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afee9a4cc7c74f85bcc21ae61fab2a2b" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34121 } }
I20260928 06:49:08.812955 25568 raft_consensus.cc:374] T 00000000000000000000000000000000 P afee9a4cc7c74f85bcc21ae61fab2a2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:08.813021 25568 raft_consensus.cc:729] T 00000000000000000000000000000000 P afee9a4cc7c74f85bcc21ae61fab2a2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afee9a4cc7c74f85bcc21ae61fab2a2b, State: Initialized, Role: FOLLOWER
I20260928 06:49:08.813539 25568 consensus_queue.cc:260] T 00000000000000000000000000000000 P afee9a4cc7c74f85bcc21ae61fab2a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afee9a4cc7c74f85bcc21ae61fab2a2b" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34121 } }
I20260928 06:49:08.813716 25568 raft_consensus.cc:388] T 00000000000000000000000000000000 P afee9a4cc7c74f85bcc21ae61fab2a2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:49:08.813796 25568 raft_consensus.cc:482] T 00000000000000000000000000000000 P afee9a4cc7c74f85bcc21ae61fab2a2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:49:08.813905 25568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afee9a4cc7c74f85bcc21ae61fab2a2b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:08.815045 25568 raft_consensus.cc:504] T 00000000000000000000000000000000 P afee9a4cc7c74f85bcc21ae61fab2a2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afee9a4cc7c74f85bcc21ae61fab2a2b" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34121 } }
I20260928 06:49:08.815397 25568 leader_election.cc:302] T 00000000000000000000000000000000 P afee9a4cc7c74f85bcc21ae61fab2a2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afee9a4cc7c74f85bcc21ae61fab2a2b; no voters:
I20260928 06:49:08.815716 25568 leader_election.cc:288] T 00000000000000000000000000000000 P afee9a4cc7c74f85bcc21ae61fab2a2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:49:08.815954 25573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afee9a4cc7c74f85bcc21ae61fab2a2b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:08.816191 25573 raft_consensus.cc:686] T 00000000000000000000000000000000 P afee9a4cc7c74f85bcc21ae61fab2a2b [term 1 LEADER]: Becoming Leader. State: Replica: afee9a4cc7c74f85bcc21ae61fab2a2b, State: Running, Role: LEADER
I20260928 06:49:08.816488 25573 consensus_queue.cc:237] T 00000000000000000000000000000000 P afee9a4cc7c74f85bcc21ae61fab2a2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afee9a4cc7c74f85bcc21ae61fab2a2b" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34121 } }
I20260928 06:49:08.816839 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:08.803076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57992 (local address 127.7.48.62:34121)
0928 06:49:08.803511 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:08.803527 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:08.803564 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:49:08.804257 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:08.804277 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:08.804618 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:08.804961 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:08.805784 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:08.806875 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:08.808224 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:08.808898 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:08.809089 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:08.811518 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:08.811524 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:08.811542 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:08.811573 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:08.813661 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:08.814208 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:08.814214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:08.814219 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:08.814351 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:08.814605 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:08.814608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:08.814610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:08.815154 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:08.815335 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:08.815543 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 06:49:08.815837 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":138,"threads_started":1}
I20260928 06:49:08.817227 25568 sys_catalog.cc:564] T 00000000000000000000000000000000 P afee9a4cc7c74f85bcc21ae61fab2a2b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:08.818310 25575 sys_catalog.cc:455] T 00000000000000000000000000000000 P afee9a4cc7c74f85bcc21ae61fab2a2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader afee9a4cc7c74f85bcc21ae61fab2a2b. Latest consensus state: current_term: 1 leader_uuid: "afee9a4cc7c74f85bcc21ae61fab2a2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afee9a4cc7c74f85bcc21ae61fab2a2b" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34121 } } }
I20260928 06:49:08.818431 25575 sys_catalog.cc:458] T 00000000000000000000000000000000 P afee9a4cc7c74f85bcc21ae61fab2a2b [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:08.818651 25574 sys_catalog.cc:455] T 00000000000000000000000000000000 P afee9a4cc7c74f85bcc21ae61fab2a2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afee9a4cc7c74f85bcc21ae61fab2a2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afee9a4cc7c74f85bcc21ae61fab2a2b" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34121 } } }
I20260928 06:49:08.818754 25574 sys_catalog.cc:458] T 00000000000000000000000000000000 P afee9a4cc7c74f85bcc21ae61fab2a2b [sys.catalog]: This master's current role is: LEADER
W20260928 06:49:08.822103 25588 catalog_manager.cc:1558] T 00000000000000000000000000000000 P afee9a4cc7c74f85bcc21ae61fab2a2b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:49:08.822182 25588 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:49:08.822315 25582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:08.823151 25582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:49:08.828819 25582 catalog_manager.cc:1347] Generated new cluster ID: 26204b328d6a4166b5ba84793b95247f
I20260928 06:49:08.828920 25582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:08.848016 25582 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:08.849103 25582 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:08.858861 25582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afee9a4cc7c74f85bcc21ae61fab2a2b: Generated new TSK 0
I20260928 06:49:08.859784 25582 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
I20260928 06:49:08.899379 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:34121
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:33417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:09.013049 25594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:09.013307 25594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:09.013350 25594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:09.017320 25594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:09.017376 25594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:09.017441 25594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:49:09.022886 25594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:33417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:34121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:09.023867 25594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:09.024856 25594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:09.034143 25599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:09.034147 25602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:09.034147 25600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:09.034428 25594 server_base.cc:1034] running on GCE node
I20260928 06:49:09.034937 25594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:09.035547 25594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:09.036723 25594 hybrid_clock.cc:648] HybridClock initialized: now 1790578149036698 us; error 45 us; skew 500 ppm
Sep 28 06:49:09 dist-test-slave-twl6 krb5kdc[25487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578149, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:09.039471 25594 init.cc:377] Logged in from keytab as kudu/127.7.48.1@KRBTEST.COM (short username kudu)
I20260928 06:49:09.040356 25594 webserver.cc:460] Webserver started at http://127.7.48.1:39593/ using document root <none> and password file <none>
I20260928 06:49:09.040895 25594 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:09.040990 25594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:09.041229 25594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:09.043202 25594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "26bc9ebe4d104c049076a45737e263eb"
format_stamp: "Formatted at 2026-09-28 06:49:09 on dist-test-slave-twl6"
server_key: "9904abbf8a4f3913d063b2506e868845"
server_key_iv: "d016534ea039f4c6a366d3b3e8ca9239"
server_key_version: "encryptionkey@0"
I20260928 06:49:09.043800 25594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "26bc9ebe4d104c049076a45737e263eb"
format_stamp: "Formatted at 2026-09-28 06:49:09 on dist-test-slave-twl6"
server_key: "9904abbf8a4f3913d063b2506e868845"
server_key_iv: "d016534ea039f4c6a366d3b3e8ca9239"
server_key_version: "encryptionkey@0"
I20260928 06:49:09.047214 25594 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:49:09.049374 25609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:09.050488 25594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:49:09.050613 25594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "26bc9ebe4d104c049076a45737e263eb"
format_stamp: "Formatted at 2026-09-28 06:49:09 on dist-test-slave-twl6"
server_key: "9904abbf8a4f3913d063b2506e868845"
server_key_iv: "d016534ea039f4c6a366d3b3e8ca9239"
server_key_version: "encryptionkey@0"
I20260928 06:49:09.050712 25594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:09.082158 25594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:09.086233 25594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:09.086476 25594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:09.087113 25594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:49:09.088289 25594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:49:09.088344 25594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:09.088419 25594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:49:09.088505 25594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:09.097198 25594 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:39987
I20260928 06:49:09.097219 25722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:39987 every 8 connection(s)
I20260928 06:49:09.098157 25594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:49:09.106688 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 25594
I20260928 06:49:09.106825 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
Sep 28 06:49:09 dist-test-slave-twl6 krb5kdc[25487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578149, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:49:09.107149 7360 external_mini_cluster.cc:1428] Setting key b32e8195a0651339fa49987a44aca26f
I20260928 06:49:09.111898 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:09.099967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:36023 (local address 127.7.48.62:34121)
0928 06:49:09.100123 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:09.100127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:09.100996 (+ 869us) server_negotiation.cc:407] Connection header received
0928 06:49:09.101863 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:09.101866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:09.101913 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:09.101996 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:09.103117 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.103662 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.104832 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.105085 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.105172 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:09.107741 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:09.107745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:09.107747 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:09.107785 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:09.109637 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:09.110129 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:09.110135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:09.110139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:09.110236 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:09.110616 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:09.110621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:09.110625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:09.110885 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:09.111056 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:09.111528 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 06:49:09.111669 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:49:09.112625 25725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:09.100396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34121 (local address 127.7.48.1:36023)
0928 06:49:09.100843 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:09.100885 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:09.101668 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:09.102160 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:09.102167 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:09.102522 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:09.102982 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:09.102994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.103769 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.103771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:09.104683 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:09.104710 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.105214 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.105217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:09.105349 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:09.105881 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:09.105896 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:09.107531 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:09.109755 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:09.109759 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:09.109771 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:09.110025 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:09.110335 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:09.110337 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:09.110339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:09.110480 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:09.111016 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:09.111024 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:09.111292 (+ 268us) client_negotiation.cc:769] Sending connection context
0928 06:49:09.111453 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 06:49:09.111690 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":115,"threads_started":1}
I20260928 06:49:09.113855 25723 heartbeater.cc:347] Connected to a master server at 127.7.48.62:34121
I20260928 06:49:09.114100 25723 heartbeater.cc:464] Registering TS with master...
I20260928 06:49:09.114570 25723 heartbeater.cc:511] Master 127.7.48.62:34121 requested a full tablet report, sending...
I20260928 06:49:09.116109 25532 ts_manager.cc:194] Registered new tserver with Master: 26bc9ebe4d104c049076a45737e263eb (127.7.48.1:39987)
I20260928 06:49:09.117249 25532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.1@KRBTEST.COM'} at 127.7.48.1:36023
WARNING: no policy specified for kudu/127.7.48.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
I20260928 06:49:09.134367 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:34121
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:33417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:09.241415 25728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:09.241660 25728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:09.241700 25728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:09.245657 25728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:09.245709 25728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:09.245762 25728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:49:09.250730 25728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:33417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:34121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:09.251775 25728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:09.252737 25728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:09.262348 25736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:09.262358 25734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:09.262357 25733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:09.262393 25728 server_base.cc:1034] running on GCE node
I20260928 06:49:09.263018 25728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:09.263633 25728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:09.264814 25728 hybrid_clock.cc:648] HybridClock initialized: now 1790578149264792 us; error 35 us; skew 500 ppm
Sep 28 06:49:09 dist-test-slave-twl6 krb5kdc[25487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578149, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:09.267475 25728 init.cc:377] Logged in from keytab as kudu/127.7.48.2@KRBTEST.COM (short username kudu)
I20260928 06:49:09.268404 25728 webserver.cc:460] Webserver started at http://127.7.48.2:33771/ using document root <none> and password file <none>
I20260928 06:49:09.268981 25728 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:09.269069 25728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:09.269292 25728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:09.271282 25728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "7c27d7d397634e5583f26c245558afc7"
format_stamp: "Formatted at 2026-09-28 06:49:09 on dist-test-slave-twl6"
server_key: "09bd71d4e7d3dd941f0c41cee1f603ee"
server_key_iv: "db60964a6a1758603425d7962a23a354"
server_key_version: "encryptionkey@0"
I20260928 06:49:09.271842 25728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "7c27d7d397634e5583f26c245558afc7"
format_stamp: "Formatted at 2026-09-28 06:49:09 on dist-test-slave-twl6"
server_key: "09bd71d4e7d3dd941f0c41cee1f603ee"
server_key_iv: "db60964a6a1758603425d7962a23a354"
server_key_version: "encryptionkey@0"
I20260928 06:49:09.275349 25728 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:49:09.277595 25743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:09.278762 25728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:49:09.278885 25728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "7c27d7d397634e5583f26c245558afc7"
format_stamp: "Formatted at 2026-09-28 06:49:09 on dist-test-slave-twl6"
server_key: "09bd71d4e7d3dd941f0c41cee1f603ee"
server_key_iv: "db60964a6a1758603425d7962a23a354"
server_key_version: "encryptionkey@0"
I20260928 06:49:09.278982 25728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:09.292906 25728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:09.296003 25728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:09.296182 25728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:09.296684 25728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:49:09.297395 25728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:49:09.297458 25728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:09.297521 25728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:49:09.297569 25728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:09.305786 25728 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:40111
I20260928 06:49:09.305809 25856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:40111 every 8 connection(s)
I20260928 06:49:09.306561 25728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:49:09.310189 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 25728
I20260928 06:49:09.310331 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:49:09.310673 7360 external_mini_cluster.cc:1428] Setting key 23975bfecdf9f7be35266be4cbdc29c4
Sep 28 06:49:09 dist-test-slave-twl6 krb5kdc[25487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578149, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.48.3@KRBTEST.COM; defaulting to no policy
I20260928 06:49:09.321609 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:09.308338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:48281 (local address 127.7.48.62:34121)
0928 06:49:09.308480 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:09.308484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:09.309230 (+ 746us) server_negotiation.cc:407] Connection header received
0928 06:49:09.310065 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:09.310067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:09.310116 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:09.310188 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:09.311637 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.312341 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.314105 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.314325 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.314437 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:09.317586 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:09.317589 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:09.317591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:09.317614 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:09.319440 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:09.320067 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:09.320077 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:09.320078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:09.320163 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:09.320544 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:09.320546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:09.320547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:09.320698 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:09.320788 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:09.321393 (+ 605us) server_negotiation.cc:299] Negotiation successful
0928 06:49:09.321490 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:49:09.322512 25859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:09.308634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34121 (local address 127.7.48.2:48281)
0928 06:49:09.309076 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:09.309116 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:09.309835 (+ 719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:09.310402 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:09.310410 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:09.310890 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:09.311496 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:09.311513 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.312468 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.312474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:09.313971 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:09.313986 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.314481 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.314485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:09.314785 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:09.315426 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:09.315441 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:09.317362 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:09.319565 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:09.319571 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:09.319586 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:09.319963 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:09.320272 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:09.320276 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:09.320279 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:09.320453 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:09.320830 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:09.320836 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:09.321139 (+ 303us) client_negotiation.cc:769] Sending connection context
0928 06:49:09.321364 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 06:49:09.321639 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":120,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.3@KRBTEST.COM" created.
I20260928 06:49:09.323719 25857 heartbeater.cc:347] Connected to a master server at 127.7.48.62:34121
I20260928 06:49:09.323959 25857 heartbeater.cc:464] Registering TS with master...
I20260928 06:49:09.324430 25857 heartbeater.cc:511] Master 127.7.48.62:34121 requested a full tablet report, sending...
I20260928 06:49:09.325528 25532 ts_manager.cc:194] Registered new tserver with Master: 7c27d7d397634e5583f26c245558afc7 (127.7.48.2:40111)
I20260928 06:49:09.326128 25532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.2@KRBTEST.COM'} at 127.7.48.2:48281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
I20260928 06:49:09.338171 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:34121
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:33417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:09.446717 25862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:09.446978 25862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:09.447021 25862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:09.450762 25862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:09.450822 25862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:09.450881 25862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:49:09.459956 25862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:33417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:34121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:09.461254 25862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:09.462277 25862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:09.471838 25867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:09.471853 25868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:09.472108 25862 server_base.cc:1034] running on GCE node
W20260928 06:49:09.471845 25870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:09.472610 25862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:09.473191 25862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:09.474346 25862 hybrid_clock.cc:648] HybridClock initialized: now 1790578149474331 us; error 27 us; skew 500 ppm
Sep 28 06:49:09 dist-test-slave-twl6 krb5kdc[25487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578149, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:09.476972 25862 init.cc:377] Logged in from keytab as kudu/127.7.48.3@KRBTEST.COM (short username kudu)
I20260928 06:49:09.477823 25862 webserver.cc:460] Webserver started at http://127.7.48.3:40629/ using document root <none> and password file <none>
I20260928 06:49:09.478399 25862 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:09.478487 25862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:09.478710 25862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:09.480726 25862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "b1e0b84b901840609c6fccdedf3b99be"
format_stamp: "Formatted at 2026-09-28 06:49:09 on dist-test-slave-twl6"
server_key: "f318127c4f08727a5916a8d592c36473"
server_key_iv: "a37b8f7f9b92f5536f70b386cf58b73a"
server_key_version: "encryptionkey@0"
I20260928 06:49:09.481269 25862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "b1e0b84b901840609c6fccdedf3b99be"
format_stamp: "Formatted at 2026-09-28 06:49:09 on dist-test-slave-twl6"
server_key: "f318127c4f08727a5916a8d592c36473"
server_key_iv: "a37b8f7f9b92f5536f70b386cf58b73a"
server_key_version: "encryptionkey@0"
I20260928 06:49:09.484939 25862 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260928 06:49:09.487186 25877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:09.488409 25862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 06:49:09.488539 25862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "b1e0b84b901840609c6fccdedf3b99be"
format_stamp: "Formatted at 2026-09-28 06:49:09 on dist-test-slave-twl6"
server_key: "f318127c4f08727a5916a8d592c36473"
server_key_iv: "a37b8f7f9b92f5536f70b386cf58b73a"
server_key_version: "encryptionkey@0"
I20260928 06:49:09.488643 25862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:09.508473 25862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:09.511312 25862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:09.511512 25862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:09.512136 25862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:49:09.512982 25862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:49:09.513053 25862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:09.513137 25862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:49:09.513193 25862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:09.522575 25862 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:45145
I20260928 06:49:09.522605 25990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:45145 every 8 connection(s)
I20260928 06:49:09.523478 25862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:49:09.526686 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 25862
I20260928 06:49:09.526811 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:49:09.527138 7360 external_mini_cluster.cc:1428] Setting key d932385665225850733c82ffb8e94e59
Sep 28 06:49:09 dist-test-slave-twl6 krb5kdc[25487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578149, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:49:09.536469 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:09.525276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:50811 (local address 127.7.48.62:34121)
0928 06:49:09.525410 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:09.525414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:09.526213 (+ 799us) server_negotiation.cc:407] Connection header received
0928 06:49:09.527351 (+ 1138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:09.527354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:09.527400 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:09.527503 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:09.528932 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.529528 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.530635 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.530847 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.530957 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:09.533292 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:09.533296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:09.533298 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:09.533319 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:09.534488 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:09.534999 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:09.535003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:09.535004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:09.535083 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:09.535403 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:09.535407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:09.535409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:09.535590 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:09.535696 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:09.536198 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 06:49:09.536308 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:49:09.537231 25993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:09.525643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:34121 (local address 127.7.48.3:50811)
0928 06:49:09.526081 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:09.526120 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:09.527142 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:09.527675 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:09.527684 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:09.528199 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:09.528799 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:09.528812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.529653 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.529656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:09.530507 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:09.530516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.530989 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.530993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:09.531159 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:09.531660 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:09.531675 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:09.533121 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:09.534636 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:09.534654 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:09.534667 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:09.534898 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:09.535179 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:09.535182 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:09.535184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:09.535311 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:09.535693 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:09.535699 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:09.535974 (+ 275us) client_negotiation.cc:769] Sending connection context
0928 06:49:09.536158 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 06:49:09.536448 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":117,"threads_started":1}
I20260928 06:49:09.538203 25991 heartbeater.cc:347] Connected to a master server at 127.7.48.62:34121
I20260928 06:49:09.538437 25991 heartbeater.cc:464] Registering TS with master...
I20260928 06:49:09.538885 25991 heartbeater.cc:511] Master 127.7.48.62:34121 requested a full tablet report, sending...
I20260928 06:49:09.539935 25532 ts_manager.cc:194] Registered new tserver with Master: b1e0b84b901840609c6fccdedf3b99be (127.7.48.3:45145)
I20260928 06:49:09.540560 25532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.3@KRBTEST.COM'} at 127.7.48.3:50811
I20260928 06:49:09.540796 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:49:09.549793 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:09.542751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57994 (local address 127.7.48.62:34121)
0928 06:49:09.542870 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:09.542873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:09.543064 (+ 191us) server_negotiation.cc:407] Connection header received
0928 06:49:09.543159 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:09.543161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:09.543195 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:09.543254 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:09.543868 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.544447 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.545232 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.545424 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.545499 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:09.546803 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:09.546806 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:09.546808 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:09.546827 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:09.548304 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:09.548727 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:09.548729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:09.548731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:09.548799 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:09.549110 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:09.549113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:09.549114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:09.549287 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:09.549419 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:09.549625 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 06:49:09.549690 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:49:09.552364 25532 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:49:09.555472 25532 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:49:09.569550 26006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:09.564263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35074 (local address 127.7.48.3:45145)
0928 06:49:09.564719 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:09.564732 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:09.564755 (+ 23us) server_negotiation.cc:407] Connection header received
0928 06:49:09.564832 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:09.564838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:09.565044 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:09.565583 (+ 539us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:09.565742 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.566447 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.567984 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.568779 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.568894 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:09.569039 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:09.569273 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 06:49:09.569389 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":158,"threads_started":1}
I20260928 06:49:09.569543 26002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:09.563554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:45145 (local address 127.0.0.1:35074)
0928 06:49:09.563893 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:09.563941 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:09.564070 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:09.565242 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:09.565248 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:09.565272 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:09.565401 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:09.565405 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.566573 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.566579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:09.567813 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:09.567823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.568912 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.568916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:09.569035 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:09.569085 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 06:49:09.569231 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 06:49:09.569392 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":3,"thread_start_us":89,"threads_started":1}
I20260928 06:49:09.569870 26007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:09.564808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57024 (local address 127.7.48.1:39987)
0928 06:49:09.565184 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:09.565199 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:09.565216 (+ 17us) server_negotiation.cc:407] Connection header received
0928 06:49:09.565274 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:09.565278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:09.565416 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:09.565862 (+ 446us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:09.566016 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.567017 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.568322 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.569076 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.569207 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:09.569340 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:09.569623 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 06:49:09.569722 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":90,"threads_started":1}
I20260928 06:49:09.570006 26005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:09.564233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40630 (local address 127.7.48.2:40111)
0928 06:49:09.564524 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:09.564536 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:09.565054 (+ 518us) server_negotiation.cc:407] Connection header received
0928 06:49:09.565147 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:09.565152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:09.565334 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:09.565485 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:09.565720 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.566850 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.568200 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.568787 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.568919 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:09.569057 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:09.569744 (+ 687us) server_negotiation.cc:299] Negotiation successful
0928 06:49:09.569834 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":123,"threads_started":1}
I20260928 06:49:09.570127 26003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:09.563645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:39987 (local address 127.0.0.1:57024)
0928 06:49:09.564284 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:09.564298 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:09.564388 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:09.565553 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:09.565556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:09.565600 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:09.565713 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:09.565720 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.567141 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.567144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:09.568201 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:09.568210 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.569218 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.569222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:09.569327 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:09.569372 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 06:49:09.569444 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 06:49:09.569509 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":95,"threads_started":1}
I20260928 06:49:09.570325 26004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:09.563862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:40111 (local address 127.0.0.1:40630)
0928 06:49:09.564854 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:09.564866 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:09.564958 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:09.565466 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:09.565472 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:09.565490 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:09.565604 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:09.565610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.566987 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.566991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:09.568073 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:09.568081 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.569440 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.569443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:09.569517 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:09.569534 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 06:49:09.569599 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:49:09.570173 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"thread_start_us":126,"threads_started":1}
I20260928 06:49:09.572834 25925 tablet_service.cc:1467] Processing CreateTablet for tablet 257c552934c64b1dab31d163f8893684 (DEFAULT_TABLE table=test-table [id=6a90f49ed84b484bbd4a57a40861b3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:49:09.572885 25657 tablet_service.cc:1467] Processing CreateTablet for tablet 257c552934c64b1dab31d163f8893684 (DEFAULT_TABLE table=test-table [id=6a90f49ed84b484bbd4a57a40861b3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:49:09.572885 25791 tablet_service.cc:1467] Processing CreateTablet for tablet 257c552934c64b1dab31d163f8893684 (DEFAULT_TABLE table=test-table [id=6a90f49ed84b484bbd4a57a40861b3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:49:09.573836 25925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 257c552934c64b1dab31d163f8893684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:09.573902 25657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 257c552934c64b1dab31d163f8893684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:09.573902 25791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 257c552934c64b1dab31d163f8893684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:09.580147 26008 tablet_bootstrap.cc:492] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be: Bootstrap starting.
I20260928 06:49:09.581032 26009 tablet_bootstrap.cc:492] T 257c552934c64b1dab31d163f8893684 P 26bc9ebe4d104c049076a45737e263eb: Bootstrap starting.
I20260928 06:49:09.582350 26008 tablet_bootstrap.cc:654] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:09.582826 26009 tablet_bootstrap.cc:654] T 257c552934c64b1dab31d163f8893684 P 26bc9ebe4d104c049076a45737e263eb: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:09.583531 26008 log.cc:824] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:09.583806 26009 log.cc:824] T 257c552934c64b1dab31d163f8893684 P 26bc9ebe4d104c049076a45737e263eb: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:09.584192 26010 tablet_bootstrap.cc:492] T 257c552934c64b1dab31d163f8893684 P 7c27d7d397634e5583f26c245558afc7: Bootstrap starting.
I20260928 06:49:09.585469 26008 tablet_bootstrap.cc:492] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be: No bootstrap required, opened a new log
I20260928 06:49:09.585580 26009 tablet_bootstrap.cc:492] T 257c552934c64b1dab31d163f8893684 P 26bc9ebe4d104c049076a45737e263eb: No bootstrap required, opened a new log
I20260928 06:49:09.585651 26008 ts_tablet_manager.cc:1397] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 06:49:09.585731 26009 ts_tablet_manager.cc:1397] T 257c552934c64b1dab31d163f8893684 P 26bc9ebe4d104c049076a45737e263eb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 06:49:09.586385 26010 tablet_bootstrap.cc:654] T 257c552934c64b1dab31d163f8893684 P 7c27d7d397634e5583f26c245558afc7: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:09.587000 26010 log.cc:824] T 257c552934c64b1dab31d163f8893684 P 7c27d7d397634e5583f26c245558afc7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:09.588550 26010 tablet_bootstrap.cc:492] T 257c552934c64b1dab31d163f8893684 P 7c27d7d397634e5583f26c245558afc7: No bootstrap required, opened a new log
I20260928 06:49:09.588740 26010 ts_tablet_manager.cc:1397] T 257c552934c64b1dab31d163f8893684 P 7c27d7d397634e5583f26c245558afc7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 06:49:09.589260 26008 raft_consensus.cc:348] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e0b84b901840609c6fccdedf3b99be" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 45145 } } peers { permanent_uuid: "26bc9ebe4d104c049076a45737e263eb" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 39987 } } peers { permanent_uuid: "7c27d7d397634e5583f26c245558afc7" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40111 } }
I20260928 06:49:09.589260 26009 raft_consensus.cc:348] T 257c552934c64b1dab31d163f8893684 P 26bc9ebe4d104c049076a45737e263eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e0b84b901840609c6fccdedf3b99be" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 45145 } } peers { permanent_uuid: "26bc9ebe4d104c049076a45737e263eb" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 39987 } } peers { permanent_uuid: "7c27d7d397634e5583f26c245558afc7" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40111 } }
I20260928 06:49:09.589529 26009 raft_consensus.cc:374] T 257c552934c64b1dab31d163f8893684 P 26bc9ebe4d104c049076a45737e263eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:09.589529 26008 raft_consensus.cc:374] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:09.589584 26008 raft_consensus.cc:729] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e0b84b901840609c6fccdedf3b99be, State: Initialized, Role: FOLLOWER
I20260928 06:49:09.589584 26009 raft_consensus.cc:729] T 257c552934c64b1dab31d163f8893684 P 26bc9ebe4d104c049076a45737e263eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26bc9ebe4d104c049076a45737e263eb, State: Initialized, Role: FOLLOWER
I20260928 06:49:09.590027 26009 consensus_queue.cc:260] T 257c552934c64b1dab31d163f8893684 P 26bc9ebe4d104c049076a45737e263eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e0b84b901840609c6fccdedf3b99be" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 45145 } } peers { permanent_uuid: "26bc9ebe4d104c049076a45737e263eb" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 39987 } } peers { permanent_uuid: "7c27d7d397634e5583f26c245558afc7" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40111 } }
I20260928 06:49:09.590026 26008 consensus_queue.cc:260] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e0b84b901840609c6fccdedf3b99be" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 45145 } } peers { permanent_uuid: "26bc9ebe4d104c049076a45737e263eb" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 39987 } } peers { permanent_uuid: "7c27d7d397634e5583f26c245558afc7" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40111 } }
I20260928 06:49:09.590860 25723 heartbeater.cc:503] Master 127.7.48.62:34121 was elected leader, sending a full tablet report...
I20260928 06:49:09.590893 25991 heartbeater.cc:503] Master 127.7.48.62:34121 was elected leader, sending a full tablet report...
I20260928 06:49:09.590969 26008 ts_tablet_manager.cc:1428] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 06:49:09.592067 26009 ts_tablet_manager.cc:1428] T 257c552934c64b1dab31d163f8893684 P 26bc9ebe4d104c049076a45737e263eb: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260928 06:49:09.593583 26010 raft_consensus.cc:348] T 257c552934c64b1dab31d163f8893684 P 7c27d7d397634e5583f26c245558afc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e0b84b901840609c6fccdedf3b99be" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 45145 } } peers { permanent_uuid: "26bc9ebe4d104c049076a45737e263eb" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 39987 } } peers { permanent_uuid: "7c27d7d397634e5583f26c245558afc7" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40111 } }
I20260928 06:49:09.593852 26010 raft_consensus.cc:374] T 257c552934c64b1dab31d163f8893684 P 7c27d7d397634e5583f26c245558afc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:09.593925 26010 raft_consensus.cc:729] T 257c552934c64b1dab31d163f8893684 P 7c27d7d397634e5583f26c245558afc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c27d7d397634e5583f26c245558afc7, State: Initialized, Role: FOLLOWER
I20260928 06:49:09.594365 26010 consensus_queue.cc:260] T 257c552934c64b1dab31d163f8893684 P 7c27d7d397634e5583f26c245558afc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e0b84b901840609c6fccdedf3b99be" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 45145 } } peers { permanent_uuid: "26bc9ebe4d104c049076a45737e263eb" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 39987 } } peers { permanent_uuid: "7c27d7d397634e5583f26c245558afc7" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40111 } }
I20260928 06:49:09.595204 26010 ts_tablet_manager.cc:1428] T 257c552934c64b1dab31d163f8893684 P 7c27d7d397634e5583f26c245558afc7: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260928 06:49:09.595527 25857 heartbeater.cc:503] Master 127.7.48.62:34121 was elected leader, sending a full tablet report...
W20260928 06:49:09.599252 25724 tablet.cc:2369] T 257c552934c64b1dab31d163f8893684 P 26bc9ebe4d104c049076a45737e263eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:49:09.602187 26014 raft_consensus.cc:482] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:49:09.602360 26014 raft_consensus.cc:504] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e0b84b901840609c6fccdedf3b99be" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 45145 } } peers { permanent_uuid: "26bc9ebe4d104c049076a45737e263eb" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 39987 } } peers { permanent_uuid: "7c27d7d397634e5583f26c245558afc7" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40111 } }
I20260928 06:49:09.603365 26014 leader_election.cc:288] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26bc9ebe4d104c049076a45737e263eb (127.7.48.1:39987), 7c27d7d397634e5583f26c245558afc7 (127.7.48.2:40111)
I20260928 06:49:09.606777 25993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:09.603516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:39987 (local address 127.7.48.3:47955)
0928 06:49:09.603678 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:09.603697 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:09.603787 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:09.604086 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:09.604090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:09.604110 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:09.604226 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:09.604235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.604947 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.604950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:09.605723 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:09.605731 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.606356 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.606361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:09.606453 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:09.606476 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 06:49:09.606540 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 06:49:09.606644 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 06:49:09.606778 26007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:09.603587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:47955 (local address 127.7.48.1:39987)
0928 06:49:09.603690 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:09.603692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:09.603761 (+ 69us) server_negotiation.cc:407] Connection header received
0928 06:49:09.603919 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:09.603922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:09.603972 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:09.604041 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:09.604325 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.604848 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.605840 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.606239 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.606311 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:09.606378 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:09.606575 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 06:49:09.606668 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:49:09.606812 26005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:09.603978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:48059 (local address 127.7.48.2:40111)
0928 06:49:09.604092 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:09.604096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:09.604107 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:49:09.604145 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:09.604147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:09.604181 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:09.604256 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:09.604501 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.604943 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.605919 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.606273 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.606395 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:09.606449 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:09.606650 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 06:49:09.606716 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:49:09.606819 26017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:09.603611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:40111 (local address 127.7.48.3:48059)
0928 06:49:09.603941 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:09.603954 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:09.604036 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:09.604270 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:09.604272 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:09.604304 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:09.604405 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:09.604410 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.605042 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.605044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:09.605806 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:09.605817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.606413 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.606416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:09.606501 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:09.606517 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 06:49:09.606631 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 06:49:09.606690 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":108,"threads_started":1}
I20260928 06:49:09.607406 25677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257c552934c64b1dab31d163f8893684" candidate_uuid: "b1e0b84b901840609c6fccdedf3b99be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26bc9ebe4d104c049076a45737e263eb" is_pre_election: true
I20260928 06:49:09.607434 25811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257c552934c64b1dab31d163f8893684" candidate_uuid: "b1e0b84b901840609c6fccdedf3b99be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c27d7d397634e5583f26c245558afc7" is_pre_election: true
I20260928 06:49:09.607678 25811 raft_consensus.cc:2454] T 257c552934c64b1dab31d163f8893684 P 7c27d7d397634e5583f26c245558afc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1e0b84b901840609c6fccdedf3b99be in term 0.
I20260928 06:49:09.607676 25677 raft_consensus.cc:2454] T 257c552934c64b1dab31d163f8893684 P 26bc9ebe4d104c049076a45737e263eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1e0b84b901840609c6fccdedf3b99be in term 0.
I20260928 06:49:09.608140 25879 leader_election.cc:302] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26bc9ebe4d104c049076a45737e263eb, b1e0b84b901840609c6fccdedf3b99be; no voters:
I20260928 06:49:09.608395 26014 raft_consensus.cc:2781] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:49:09.608480 26014 raft_consensus.cc:482] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:49:09.608526 26014 raft_consensus.cc:3037] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:09.609479 26014 raft_consensus.cc:504] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e0b84b901840609c6fccdedf3b99be" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 45145 } } peers { permanent_uuid: "26bc9ebe4d104c049076a45737e263eb" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 39987 } } peers { permanent_uuid: "7c27d7d397634e5583f26c245558afc7" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40111 } }
I20260928 06:49:09.609773 26014 leader_election.cc:288] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be [CANDIDATE]: Term 1 election: Requested vote from peers 26bc9ebe4d104c049076a45737e263eb (127.7.48.1:39987), 7c27d7d397634e5583f26c245558afc7 (127.7.48.2:40111)
I20260928 06:49:09.610074 25811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257c552934c64b1dab31d163f8893684" candidate_uuid: "b1e0b84b901840609c6fccdedf3b99be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c27d7d397634e5583f26c245558afc7"
I20260928 06:49:09.610072 25677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257c552934c64b1dab31d163f8893684" candidate_uuid: "b1e0b84b901840609c6fccdedf3b99be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26bc9ebe4d104c049076a45737e263eb"
I20260928 06:49:09.610188 25677 raft_consensus.cc:3037] T 257c552934c64b1dab31d163f8893684 P 26bc9ebe4d104c049076a45737e263eb [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:09.610188 25811 raft_consensus.cc:3037] T 257c552934c64b1dab31d163f8893684 P 7c27d7d397634e5583f26c245558afc7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:09.611366 25811 raft_consensus.cc:2454] T 257c552934c64b1dab31d163f8893684 P 7c27d7d397634e5583f26c245558afc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1e0b84b901840609c6fccdedf3b99be in term 1.
I20260928 06:49:09.611366 25677 raft_consensus.cc:2454] T 257c552934c64b1dab31d163f8893684 P 26bc9ebe4d104c049076a45737e263eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1e0b84b901840609c6fccdedf3b99be in term 1.
I20260928 06:49:09.611667 25879 leader_election.cc:302] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26bc9ebe4d104c049076a45737e263eb, b1e0b84b901840609c6fccdedf3b99be; no voters:
I20260928 06:49:09.611811 26014 raft_consensus.cc:2781] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:09.612051 26014 raft_consensus.cc:686] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be [term 1 LEADER]: Becoming Leader. State: Replica: b1e0b84b901840609c6fccdedf3b99be, State: Running, Role: LEADER
I20260928 06:49:09.612365 26014 consensus_queue.cc:237] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e0b84b901840609c6fccdedf3b99be" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 45145 } } peers { permanent_uuid: "26bc9ebe4d104c049076a45737e263eb" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 39987 } } peers { permanent_uuid: "7c27d7d397634e5583f26c245558afc7" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40111 } }
I20260928 06:49:09.615471 25531 catalog_manager.cc:5495] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be reported cstate change: term changed from 0 to 1, leader changed from <none> to b1e0b84b901840609c6fccdedf3b99be (127.7.48.3). New cstate: current_term: 1 leader_uuid: "b1e0b84b901840609c6fccdedf3b99be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e0b84b901840609c6fccdedf3b99be" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 45145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26bc9ebe4d104c049076a45737e263eb" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 39987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c27d7d397634e5583f26c245558afc7" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40111 } health_report { overall_health: UNKNOWN } } }
I20260928 06:49:09.631318 26006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:09.628351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.7.48.3:45145)
0928 06:49:09.628460 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:09.628463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:09.628475 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:49:09.628574 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:09.628576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:09.628625 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:09.628708 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:49:09.628908 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.629339 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.630097 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.630272 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.630337 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:09.630592 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:49:09.630940 (+ 348us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:49:09.631040 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:09.631112 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 06:49:09.631142 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:49:09.636845 25811 raft_consensus.cc:1261] T 257c552934c64b1dab31d163f8893684 P 7c27d7d397634e5583f26c245558afc7 [term 1 FOLLOWER]: Refusing update from remote peer b1e0b84b901840609c6fccdedf3b99be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:49:09.636844 25677 raft_consensus.cc:1261] T 257c552934c64b1dab31d163f8893684 P 26bc9ebe4d104c049076a45737e263eb [term 1 FOLLOWER]: Refusing update from remote peer b1e0b84b901840609c6fccdedf3b99be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:49:09.637487 26018 consensus_queue.cc:1035] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be [LEADER]: Connected to new peer: Peer: permanent_uuid: "26bc9ebe4d104c049076a45737e263eb" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 39987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:49:09.637646 26014 consensus_queue.cc:1035] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c27d7d397634e5583f26c245558afc7" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:49:09.646308 26023 mvcc.cc:205] Tried to move back new op lower bound from 7334208100906090496 to 7334208100814700544. Current Snapshot: MvccSnapshot[applied={T|T < 7334208100906090496}]
I20260928 06:49:09.652807 25531 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:49:09.652979 25531 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:49:09.655440 25531 catalog_manager.cc:5782] T 00000000000000000000000000000000 P afee9a4cc7c74f85bcc21ae61fab2a2b: Sending DeleteTablet for 3 replicas of tablet 257c552934c64b1dab31d163f8893684
I20260928 06:49:09.656219 25657 tablet_service.cc:1514] Processing DeleteTablet for tablet 257c552934c64b1dab31d163f8893684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:49:09 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:57024
I20260928 06:49:09.656399 25925 tablet_service.cc:1514] Processing DeleteTablet for tablet 257c552934c64b1dab31d163f8893684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:49:09 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:35074
I20260928 06:49:09.656800 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 25594
I20260928 06:49:09.656839 25791 tablet_service.cc:1514] Processing DeleteTablet for tablet 257c552934c64b1dab31d163f8893684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:49:09 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:40630
I20260928 06:49:09.658649 26032 tablet_replica.cc:330] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be: stopping tablet replica
I20260928 06:49:09.658990 26032 raft_consensus.cc:2229] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:49:09.659577 26032 raft_consensus.cc:2258] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:49:09.660039 26033 tablet_replica.cc:330] T 257c552934c64b1dab31d163f8893684 P 7c27d7d397634e5583f26c245558afc7: stopping tablet replica
I20260928 06:49:09.660349 26033 raft_consensus.cc:2229] T 257c552934c64b1dab31d163f8893684 P 7c27d7d397634e5583f26c245558afc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:49:09.660648 26033 raft_consensus.cc:2258] T 257c552934c64b1dab31d163f8893684 P 7c27d7d397634e5583f26c245558afc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:49:09.661281 26032 ts_tablet_manager.cc:1905] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:49:09.661904 26033 ts_tablet_manager.cc:1905] T 257c552934c64b1dab31d163f8893684 P 7c27d7d397634e5583f26c245558afc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:49:09.664172 26032 ts_tablet_manager.cc:1918] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:49:09.664332 26032 log.cc:1196] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/257c552934c64b1dab31d163f8893684
I20260928 06:49:09.664649 26033 ts_tablet_manager.cc:1918] T 257c552934c64b1dab31d163f8893684 P 7c27d7d397634e5583f26c245558afc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:49:09.664746 26033 log.cc:1196] T 257c552934c64b1dab31d163f8893684 P 7c27d7d397634e5583f26c245558afc7: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/257c552934c64b1dab31d163f8893684
I20260928 06:49:09.664855 26032 ts_tablet_manager.cc:1939] T 257c552934c64b1dab31d163f8893684 P b1e0b84b901840609c6fccdedf3b99be: Deleting consensus metadata
I20260928 06:49:09.665063 26033 ts_tablet_manager.cc:1939] T 257c552934c64b1dab31d163f8893684 P 7c27d7d397634e5583f26c245558afc7: Deleting consensus metadata
W20260928 06:49:09.666033 25517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.48.1:39987 (error 108)
I20260928 06:49:09.666021 25516 catalog_manager.cc:4841] TS 7c27d7d397634e5583f26c245558afc7 (127.7.48.2:40111): tablet 257c552934c64b1dab31d163f8893684 (table test-table [id=6a90f49ed84b484bbd4a57a40861b3b3]) successfully deleted
I20260928 06:49:09.666121 25518 catalog_manager.cc:4841] TS b1e0b84b901840609c6fccdedf3b99be (127.7.48.3:45145): tablet 257c552934c64b1dab31d163f8893684 (table test-table [id=6a90f49ed84b484bbd4a57a40861b3b3]) successfully deleted
I20260928 06:49:09.666852 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 25728
I20260928 06:49:09.667102 26003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:09.666793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:39987 (local address 127.0.0.1:57036)
0928 06:49:09.666957 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:09.667028 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.1:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 06:49:09.672525 25517 catalog_manager.cc:4584] TS 26bc9ebe4d104c049076a45737e263eb (127.7.48.1:39987): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 257c552934c64b1dab31d163f8893684: Network error: Client connection negotiation failed: client connection to 127.7.48.1:39987: connect: Connection refused (error 111)
I20260928 06:49:09.672968 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 25862
I20260928 06:49:09.679080 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 25500
2026-09-28T06:49:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2200 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T06:49:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:09Z Disabled control of system clock
I20260928 06:49:09.708907 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:37969
--webserver_interface=127.7.48.62
--webserver_port=0
--builtin_ntp_servers=127.7.48.20:41975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:37969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:49:09.815466 26038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:09.815702 26038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:09.815735 26038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:09.819483 26038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:09.819537 26038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:09.819551 26038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:09.819567 26038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:09.819581 26038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:09.824323 26038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:41975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:37969
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:37969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:09.825279 26038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:09.826545 26038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:09.834656 26044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:09.834654 26043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:09.834666 26046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:09.835155 26038 server_base.cc:1034] running on GCE node
I20260928 06:49:09.835655 26038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:09.836735 26038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:09.837917 26038 hybrid_clock.cc:648] HybridClock initialized: now 1790578149837894 us; error 40 us; skew 500 ppm
I20260928 06:49:09.839668 26038 webserver.cc:460] Webserver started at http://127.7.48.62:36565/ using document root <none> and password file <none>
I20260928 06:49:09.840288 26038 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:09.840381 26038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:09.840602 26038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:09.842645 26038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "29d80d654967407d8c2ff17833ba2875"
format_stamp: "Formatted at 2026-09-28 06:49:09 on dist-test-slave-twl6"
server_key: "2e28fa3322d6033cbc95290e8f764761"
server_key_iv: "a285d00c481824634f082892f0417f21"
server_key_version: "encryptionkey@0"
I20260928 06:49:09.843195 26038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "29d80d654967407d8c2ff17833ba2875"
format_stamp: "Formatted at 2026-09-28 06:49:09 on dist-test-slave-twl6"
server_key: "2e28fa3322d6033cbc95290e8f764761"
server_key_iv: "a285d00c481824634f082892f0417f21"
server_key_version: "encryptionkey@0"
I20260928 06:49:09.846727 26038 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 06:49:09.849104 26052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:09.850274 26038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:49:09.850417 26038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "29d80d654967407d8c2ff17833ba2875"
format_stamp: "Formatted at 2026-09-28 06:49:09 on dist-test-slave-twl6"
server_key: "2e28fa3322d6033cbc95290e8f764761"
server_key_iv: "a285d00c481824634f082892f0417f21"
server_key_version: "encryptionkey@0"
I20260928 06:49:09.850521 26038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:09.871065 26038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:09.871814 26038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:09.872028 26038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:09.878927 26038 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:37969
I20260928 06:49:09.878950 26104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:37969 every 8 connection(s)
I20260928 06:49:09.879817 26038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:09.882751 26105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:09.884179 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 26038
I20260928 06:49:09.884328 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:09.884613 7360 external_mini_cluster.cc:1428] Setting key 0402d01908fc291696bf0324a55c6d4b
I20260928 06:49:09.888818 26105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29d80d654967407d8c2ff17833ba2875: Bootstrap starting.
I20260928 06:49:09.890902 26105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29d80d654967407d8c2ff17833ba2875: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:09.891619 26105 log.cc:824] T 00000000000000000000000000000000 P 29d80d654967407d8c2ff17833ba2875: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:09.893543 26105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29d80d654967407d8c2ff17833ba2875: No bootstrap required, opened a new log
I20260928 06:49:09.893589 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:09.886014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50340 (local address 127.7.48.62:37969)
0928 06:49:09.886644 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:09.886659 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:09.886694 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:49:09.887326 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:09.887343 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:09.887717 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:09.888077 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:09.888505 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.889519 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.890399 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:09.890853 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:09.891037 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:09.891967 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:09.891971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:09.891983 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:09.892368 (+ 385us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:09.892421 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:09.892440 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:09.892552 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:09.892669 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 06:49:09.892913 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":123,"threads_started":1}
I20260928 06:49:09.897482 26105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29d80d654967407d8c2ff17833ba2875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d80d654967407d8c2ff17833ba2875" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 37969 } }
I20260928 06:49:09.897710 26105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29d80d654967407d8c2ff17833ba2875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:09.897796 26105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29d80d654967407d8c2ff17833ba2875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d80d654967407d8c2ff17833ba2875, State: Initialized, Role: FOLLOWER
I20260928 06:49:09.898211 26105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29d80d654967407d8c2ff17833ba2875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d80d654967407d8c2ff17833ba2875" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 37969 } }
I20260928 06:49:09.898358 26105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29d80d654967407d8c2ff17833ba2875 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:49:09.898440 26105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29d80d654967407d8c2ff17833ba2875 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:49:09.898552 26105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29d80d654967407d8c2ff17833ba2875 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:09.899353 26105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29d80d654967407d8c2ff17833ba2875 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d80d654967407d8c2ff17833ba2875" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 37969 } }
I20260928 06:49:09.899668 26105 leader_election.cc:302] T 00000000000000000000000000000000 P 29d80d654967407d8c2ff17833ba2875 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29d80d654967407d8c2ff17833ba2875; no voters:
I20260928 06:49:09.899989 26105 leader_election.cc:288] T 00000000000000000000000000000000 P 29d80d654967407d8c2ff17833ba2875 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:49:09.900079 26110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29d80d654967407d8c2ff17833ba2875 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:09.900347 26110 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29d80d654967407d8c2ff17833ba2875 [term 1 LEADER]: Becoming Leader. State: Replica: 29d80d654967407d8c2ff17833ba2875, State: Running, Role: LEADER
I20260928 06:49:09.900664 26110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29d80d654967407d8c2ff17833ba2875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d80d654967407d8c2ff17833ba2875" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 37969 } }
I20260928 06:49:09.901160 26105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29d80d654967407d8c2ff17833ba2875 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:09.902262 26112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29d80d654967407d8c2ff17833ba2875 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29d80d654967407d8c2ff17833ba2875. Latest consensus state: current_term: 1 leader_uuid: "29d80d654967407d8c2ff17833ba2875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d80d654967407d8c2ff17833ba2875" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 37969 } } }
I20260928 06:49:09.902384 26112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29d80d654967407d8c2ff17833ba2875 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:09.902652 26111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29d80d654967407d8c2ff17833ba2875 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29d80d654967407d8c2ff17833ba2875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d80d654967407d8c2ff17833ba2875" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 37969 } } }
I20260928 06:49:09.902745 26111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29d80d654967407d8c2ff17833ba2875 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:09.902859 26120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:09.905664 26120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:49:09.911211 26120 catalog_manager.cc:1347] Generated new cluster ID: 0452841b2aee412c9f96746c1d69d20e
I20260928 06:49:09.911289 26120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:09.947983 26120 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:09.948941 26120 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:09.953665 26120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29d80d654967407d8c2ff17833ba2875: Generated new TSK 0
I20260928 06:49:09.954375 26120 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:49:09.956934 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:37969
--builtin_ntp_servers=127.7.48.20:41975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:49:10.064828 26130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:10.065052 26130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:10.065104 26130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:10.068853 26130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:10.068895 26130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:10.068945 26130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:49:10.073522 26130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:41975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:37969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:10.074481 26130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:10.075773 26130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:10.085120 26135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:10.085114 26138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:10.085114 26136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:10.085466 26130 server_base.cc:1034] running on GCE node
I20260928 06:49:10.085852 26130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:10.086463 26130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:10.087677 26130 hybrid_clock.cc:648] HybridClock initialized: now 1790578150087643 us; error 42 us; skew 500 ppm
I20260928 06:49:10.089771 26130 webserver.cc:460] Webserver started at http://127.7.48.1:36545/ using document root <none> and password file <none>
I20260928 06:49:10.090422 26130 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:10.090482 26130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:10.090693 26130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:10.092762 26130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "68366791680f4b03a2706563ed012255"
format_stamp: "Formatted at 2026-09-28 06:49:10 on dist-test-slave-twl6"
server_key: "90918a475848576dc2d0e08b32042ead"
server_key_iv: "1d1ffee72acd46c8a7d705f412a288fd"
server_key_version: "encryptionkey@0"
I20260928 06:49:10.093298 26130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "68366791680f4b03a2706563ed012255"
format_stamp: "Formatted at 2026-09-28 06:49:10 on dist-test-slave-twl6"
server_key: "90918a475848576dc2d0e08b32042ead"
server_key_iv: "1d1ffee72acd46c8a7d705f412a288fd"
server_key_version: "encryptionkey@0"
I20260928 06:49:10.096812 26130 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:49:10.099066 26144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:10.100258 26130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:49:10.100358 26130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "68366791680f4b03a2706563ed012255"
format_stamp: "Formatted at 2026-09-28 06:49:10 on dist-test-slave-twl6"
server_key: "90918a475848576dc2d0e08b32042ead"
server_key_iv: "1d1ffee72acd46c8a7d705f412a288fd"
server_key_version: "encryptionkey@0"
I20260928 06:49:10.100481 26130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:10.112785 26130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:10.113411 26130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:10.113626 26130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:10.114183 26130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:49:10.115082 26130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:49:10.115127 26130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:10.115191 26130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:49:10.115221 26130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:10.123842 26130 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:38931
I20260928 06:49:10.123865 26257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:38931 every 8 connection(s)
I20260928 06:49:10.124835 26130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:49:10.133021 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 26130
I20260928 06:49:10.133162 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
I20260928 06:49:10.133472 7360 external_mini_cluster.cc:1428] Setting key babba06d72627d47e8facaa1182e0487
I20260928 06:49:10.133634 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:10.126612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:56895 (local address 127.7.48.62:37969)
0928 06:49:10.126753 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:10.126756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:10.127738 (+ 982us) server_negotiation.cc:407] Connection header received
0928 06:49:10.129116 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:10.129118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:10.129167 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:10.129254 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:10.130068 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.130585 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.131649 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.131835 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.131973 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:10.133055 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:10.133058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:10.133060 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:10.133078 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:10.133117 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:10.133130 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:10.133229 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:10.133410 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 06:49:10.133491 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:49:10.134543 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:10.126974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:37969 (local address 127.7.48.1:56895)
0928 06:49:10.127585 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:10.127623 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:10.128922 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:10.129435 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:10.129441 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:10.129516 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:10.129957 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:10.129968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.130685 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.130687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:10.131550 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:10.131558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.132031 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.132034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:10.132161 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:10.132722 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:49:10.132743 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:10.132799 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:49:10.132882 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:10.133256 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:10.133268 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:10.133278 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 06:49:10.133405 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 06:49:10.133676 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":109,"threads_started":1}
I20260928 06:49:10.135529 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:37969
--builtin_ntp_servers=127.7.48.20:41975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:49:10.135838 26258 heartbeater.cc:347] Connected to a master server at 127.7.48.62:37969
I20260928 06:49:10.136116 26258 heartbeater.cc:464] Registering TS with master...
I20260928 06:49:10.136593 26258 heartbeater.cc:511] Master 127.7.48.62:37969 requested a full tablet report, sending...
I20260928 06:49:10.138001 26069 ts_manager.cc:194] Registered new tserver with Master: 68366791680f4b03a2706563ed012255 (127.7.48.1:38931)
I20260928 06:49:10.139088 26069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.48.1:56895
W20260928 06:49:10.243688 26261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:10.243983 26261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:10.244083 26261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:10.247752 26261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:10.247799 26261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:10.247866 26261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:49:10.252207 26261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:41975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:37969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:10.253252 26261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:10.254751 26261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:10.264142 26267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:10.264151 26266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:10.264218 26261 server_base.cc:1034] running on GCE node
W20260928 06:49:10.264159 26269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:10.264765 26261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:10.265304 26261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:10.266458 26261 hybrid_clock.cc:648] HybridClock initialized: now 1790578150266431 us; error 46 us; skew 500 ppm
I20260928 06:49:10.268365 26261 webserver.cc:460] Webserver started at http://127.7.48.2:39035/ using document root <none> and password file <none>
I20260928 06:49:10.268993 26261 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:10.269054 26261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:10.269260 26261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:10.271258 26261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "dd6e2bc7be054d07b458b2942838dd27"
format_stamp: "Formatted at 2026-09-28 06:49:10 on dist-test-slave-twl6"
server_key: "2913ef67ff60d606faded5b3fcf1484f"
server_key_iv: "603d3d9d19f22325108e72ecf9298965"
server_key_version: "encryptionkey@0"
I20260928 06:49:10.271836 26261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "dd6e2bc7be054d07b458b2942838dd27"
format_stamp: "Formatted at 2026-09-28 06:49:10 on dist-test-slave-twl6"
server_key: "2913ef67ff60d606faded5b3fcf1484f"
server_key_iv: "603d3d9d19f22325108e72ecf9298965"
server_key_version: "encryptionkey@0"
I20260928 06:49:10.275401 26261 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:49:10.277786 26275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:10.278957 26261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:49:10.279072 26261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "dd6e2bc7be054d07b458b2942838dd27"
format_stamp: "Formatted at 2026-09-28 06:49:10 on dist-test-slave-twl6"
server_key: "2913ef67ff60d606faded5b3fcf1484f"
server_key_iv: "603d3d9d19f22325108e72ecf9298965"
server_key_version: "encryptionkey@0"
I20260928 06:49:10.279173 26261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:10.306056 26261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:10.306734 26261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:10.306937 26261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:10.307500 26261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:49:10.308439 26261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:49:10.308486 26261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:10.308548 26261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:49:10.308590 26261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:10.316993 26261 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:36855
I20260928 06:49:10.317051 26388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:36855 every 8 connection(s)
I20260928 06:49:10.317870 26261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:49:10.321655 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 26261
I20260928 06:49:10.321791 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:49:10.322157 7360 external_mini_cluster.cc:1428] Setting key 0339c54dd54afc2cd0f4ff99d6db6265
I20260928 06:49:10.324256 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:37969
--builtin_ntp_servers=127.7.48.20:41975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:49:10.327200 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:10.319555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:43293 (local address 127.7.48.62:37969)
0928 06:49:10.319692 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:10.319696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:10.320617 (+ 921us) server_negotiation.cc:407] Connection header received
0928 06:49:10.321986 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:10.321990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:10.322058 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:10.322162 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:10.323224 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.323945 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.325120 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.325511 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.325624 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:10.326700 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:10.326704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:10.326706 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:10.326718 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:10.326744 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:10.326751 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:10.326821 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:10.326973 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 06:49:10.327068 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:49:10.328122 26391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:10.319886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:37969 (local address 127.7.48.2:43293)
0928 06:49:10.320471 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:10.320508 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:10.321698 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:10.322352 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:10.322358 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:10.322438 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:10.323080 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:10.323099 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.324067 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.324076 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:10.324992 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:10.325001 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.325660 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.325664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:10.325825 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:10.326461 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:49:10.326463 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:10.326501 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:49:10.326558 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:10.326835 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:10.326845 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:10.326856 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 06:49:10.326970 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 06:49:10.327243 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":101,"threads_started":1}
I20260928 06:49:10.329186 26389 heartbeater.cc:347] Connected to a master server at 127.7.48.62:37969
I20260928 06:49:10.329420 26389 heartbeater.cc:464] Registering TS with master...
I20260928 06:49:10.329903 26389 heartbeater.cc:511] Master 127.7.48.62:37969 requested a full tablet report, sending...
I20260928 06:49:10.330940 26069 ts_manager.cc:194] Registered new tserver with Master: dd6e2bc7be054d07b458b2942838dd27 (127.7.48.2:36855)
I20260928 06:49:10.331773 26069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.48.2:43293
W20260928 06:49:10.433857 26392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:10.434134 26392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:10.434175 26392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:10.438271 26392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:10.438331 26392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:10.438390 26392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:49:10.443248 26392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:41975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:37969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:10.444406 26392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:10.445796 26392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:10.455832 26398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:10.455842 26400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:10.455936 26392 server_base.cc:1034] running on GCE node
W20260928 06:49:10.455832 26397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:10.456480 26392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:10.457120 26392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:10.458304 26392 hybrid_clock.cc:648] HybridClock initialized: now 1790578150458293 us; error 34 us; skew 500 ppm
I20260928 06:49:10.460398 26392 webserver.cc:460] Webserver started at http://127.7.48.3:34229/ using document root <none> and password file <none>
I20260928 06:49:10.460995 26392 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:10.461100 26392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:10.461335 26392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:10.463414 26392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "03878870eb15418a8c1748156b1945f7"
format_stamp: "Formatted at 2026-09-28 06:49:10 on dist-test-slave-twl6"
server_key: "af93f5c55ae8cee1029ed5cbcf8d19f5"
server_key_iv: "9524006f89325d4d7178cc4a8add8ae7"
server_key_version: "encryptionkey@0"
I20260928 06:49:10.464043 26392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "03878870eb15418a8c1748156b1945f7"
format_stamp: "Formatted at 2026-09-28 06:49:10 on dist-test-slave-twl6"
server_key: "af93f5c55ae8cee1029ed5cbcf8d19f5"
server_key_iv: "9524006f89325d4d7178cc4a8add8ae7"
server_key_version: "encryptionkey@0"
I20260928 06:49:10.467665 26392 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:49:10.470158 26406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:10.471405 26392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:49:10.471539 26392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "03878870eb15418a8c1748156b1945f7"
format_stamp: "Formatted at 2026-09-28 06:49:10 on dist-test-slave-twl6"
server_key: "af93f5c55ae8cee1029ed5cbcf8d19f5"
server_key_iv: "9524006f89325d4d7178cc4a8add8ae7"
server_key_version: "encryptionkey@0"
I20260928 06:49:10.471658 26392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:10.482589 26392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:10.483323 26392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:10.483507 26392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:10.484140 26392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:49:10.485051 26392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:49:10.485128 26392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:10.485198 26392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:49:10.485260 26392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:10.499475 26392 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:34919
I20260928 06:49:10.499495 26519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:34919 every 8 connection(s)
I20260928 06:49:10.500463 26392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:49:10.504861 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 26392
I20260928 06:49:10.504980 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:49:10.505290 7360 external_mini_cluster.cc:1428] Setting key 85b9dfef70c2e4cb28b4ffe1e5a733df
I20260928 06:49:10.510373 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:10.502427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:36863 (local address 127.7.48.62:37969)
0928 06:49:10.502592 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:10.502596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:10.503743 (+ 1147us) server_negotiation.cc:407] Connection header received
0928 06:49:10.505495 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:10.505499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:10.505572 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:10.505685 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:10.506617 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.507216 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.508535 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.508750 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.508837 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:10.509957 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:10.509960 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:10.509962 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:10.509973 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:10.509992 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:10.510000 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:10.510059 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:10.510210 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 06:49:10.510262 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:49:10.511198 26522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:10.502892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:37969 (local address 127.7.48.3:36863)
0928 06:49:10.503579 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:10.503634 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:10.505212 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:10.505865 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:10.505872 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:10.505980 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:10.506476 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:10.506489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.507332 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.507335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:10.508411 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:10.508421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.508881 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.508884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:10.509019 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:10.509711 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:49:10.509713 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:10.509752 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:49:10.509815 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:10.510081 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:10.510087 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:10.510092 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:49:10.510183 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 06:49:10.510396 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":105,"threads_started":1}
I20260928 06:49:10.512378 26520 heartbeater.cc:347] Connected to a master server at 127.7.48.62:37969
I20260928 06:49:10.512631 26520 heartbeater.cc:464] Registering TS with master...
I20260928 06:49:10.513160 26520 heartbeater.cc:511] Master 127.7.48.62:37969 requested a full tablet report, sending...
I20260928 06:49:10.514185 26069 ts_manager.cc:194] Registered new tserver with Master: 03878870eb15418a8c1748156b1945f7 (127.7.48.3:34919)
I20260928 06:49:10.514724 26069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.48.3:36863
I20260928 06:49:10.519726 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:49:10.526588 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:10.522119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50352 (local address 127.7.48.62:37969)
0928 06:49:10.522253 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:10.522257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:10.522474 (+ 217us) server_negotiation.cc:407] Connection header received
0928 06:49:10.522614 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:10.522617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:10.522666 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:10.522753 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:10.523062 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.523597 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.524889 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.525107 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.525212 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:10.526087 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:10.526090 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:10.526093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:10.526105 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:10.526129 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:10.526137 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:10.526259 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:10.526370 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 06:49:10.526407 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:49:10.529541 26069 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:49:10.532620 26069 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:49:10.547087 26533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:10.541097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39542 (local address 127.7.48.1:38931)
0928 06:49:10.542067 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:10.542083 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:10.542109 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:49:10.542248 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:10.542254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:10.542516 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:10.542681 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:10.542998 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.544313 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.545905 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.546426 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.546552 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:10.546766 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:10.546820 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 06:49:10.546912 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":862,"thread_start_us":106,"threads_started":1}
I20260928 06:49:10.547089 26532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:10.540928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:38931 (local address 127.0.0.1:39542)
0928 06:49:10.542031 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:10.542045 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:10.542137 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:10.542683 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:10.542689 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:10.542711 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:10.542873 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:10.542881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.544559 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.544563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:10.545777 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:10.545787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.546552 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.546556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:10.546660 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:10.546695 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 06:49:10.546797 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 06:49:10.546913 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1044,"mutex_wait_us":41,"thread_start_us":112,"threads_started":1}
I20260928 06:49:10.547349 26531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:10.540834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:34919 (local address 127.0.0.1:50784)
0928 06:49:10.541188 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:10.541370 (+ 182us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:10.541499 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:10.543140 (+ 1641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:10.543144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:10.543159 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:10.543259 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:10.543266 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.544681 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.544685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:10.545749 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:10.545759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.546809 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.546812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:10.546899 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:10.546922 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 06:49:10.546974 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:49:10.547050 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":70,"thread_start_us":114,"threads_started":1}
I20260928 06:49:10.547781 26534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:10.541566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50784 (local address 127.7.48.3:34919)
0928 06:49:10.542691 (+ 1125us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:10.542702 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:10.542723 (+ 21us) server_negotiation.cc:407] Connection header received
0928 06:49:10.542795 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:10.542803 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:10.543000 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:10.543154 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:10.543382 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.544547 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.545901 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.546665 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.547162 (+ 497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:10.547345 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:10.547408 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 06:49:10.547571 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1026,"thread_start_us":171,"threads_started":1}
I20260928 06:49:10.548839 26535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:10.541685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:36855 (local address 127.0.0.1:52888)
0928 06:49:10.542348 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:10.542360 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:10.542443 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:10.543041 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:10.543042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:10.543057 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:10.543169 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:10.543176 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.544225 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.544232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:10.545720 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:10.545733 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.547044 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.547047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:10.547143 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:10.547164 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 06:49:10.548658 (+ 1494us) client_negotiation.cc:240] Negotiation successful
0928 06:49:10.548742 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":48,"thread_start_us":84,"threads_started":1}
I20260928 06:49:10.548947 26536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:10.541891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52888 (local address 127.7.48.2:36855)
0928 06:49:10.542187 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:10.542198 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:10.542427 (+ 229us) server_negotiation.cc:407] Connection header received
0928 06:49:10.542537 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:10.542541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:10.542741 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:10.542889 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:10.543297 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.544120 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.545850 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.546665 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.546784 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:10.546985 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:10.548695 (+ 1710us) server_negotiation.cc:299] Negotiation successful
0928 06:49:10.548794 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":122,"threads_started":1}
I20260928 06:49:10.550729 26192 tablet_service.cc:1467] Processing CreateTablet for tablet 8fd268d0e6c948ed9c3a1772329385b6 (DEFAULT_TABLE table=test-table [id=9031243c52d346e4a8cd527c81a9e89d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:49:10.551710 26192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fd268d0e6c948ed9c3a1772329385b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:10.551257 26323 tablet_service.cc:1467] Processing CreateTablet for tablet 8fd268d0e6c948ed9c3a1772329385b6 (DEFAULT_TABLE table=test-table [id=9031243c52d346e4a8cd527c81a9e89d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:49:10.552384 26323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fd268d0e6c948ed9c3a1772329385b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:10.553144 26454 tablet_service.cc:1467] Processing CreateTablet for tablet 8fd268d0e6c948ed9c3a1772329385b6 (DEFAULT_TABLE table=test-table [id=9031243c52d346e4a8cd527c81a9e89d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:49:10.554388 26454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fd268d0e6c948ed9c3a1772329385b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:10.559481 26538 tablet_bootstrap.cc:492] T 8fd268d0e6c948ed9c3a1772329385b6 P dd6e2bc7be054d07b458b2942838dd27: Bootstrap starting.
I20260928 06:49:10.561074 26539 tablet_bootstrap.cc:492] T 8fd268d0e6c948ed9c3a1772329385b6 P 03878870eb15418a8c1748156b1945f7: Bootstrap starting.
I20260928 06:49:10.561630 26538 tablet_bootstrap.cc:654] T 8fd268d0e6c948ed9c3a1772329385b6 P dd6e2bc7be054d07b458b2942838dd27: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:10.561669 26537 tablet_bootstrap.cc:492] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255: Bootstrap starting.
I20260928 06:49:10.562539 26538 log.cc:824] T 8fd268d0e6c948ed9c3a1772329385b6 P dd6e2bc7be054d07b458b2942838dd27: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:10.563249 26539 tablet_bootstrap.cc:654] T 8fd268d0e6c948ed9c3a1772329385b6 P 03878870eb15418a8c1748156b1945f7: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:10.563746 26537 tablet_bootstrap.cc:654] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:10.564203 26539 log.cc:824] T 8fd268d0e6c948ed9c3a1772329385b6 P 03878870eb15418a8c1748156b1945f7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:10.564611 26537 log.cc:824] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:10.564682 26538 tablet_bootstrap.cc:492] T 8fd268d0e6c948ed9c3a1772329385b6 P dd6e2bc7be054d07b458b2942838dd27: No bootstrap required, opened a new log
I20260928 06:49:10.564870 26538 ts_tablet_manager.cc:1397] T 8fd268d0e6c948ed9c3a1772329385b6 P dd6e2bc7be054d07b458b2942838dd27: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 06:49:10.566037 26539 tablet_bootstrap.cc:492] T 8fd268d0e6c948ed9c3a1772329385b6 P 03878870eb15418a8c1748156b1945f7: No bootstrap required, opened a new log
I20260928 06:49:10.566256 26539 ts_tablet_manager.cc:1397] T 8fd268d0e6c948ed9c3a1772329385b6 P 03878870eb15418a8c1748156b1945f7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 06:49:10.566565 26537 tablet_bootstrap.cc:492] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255: No bootstrap required, opened a new log
I20260928 06:49:10.566747 26537 ts_tablet_manager.cc:1397] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 06:49:10.569316 26538 raft_consensus.cc:348] T 8fd268d0e6c948ed9c3a1772329385b6 P dd6e2bc7be054d07b458b2942838dd27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03878870eb15418a8c1748156b1945f7" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34919 } } peers { permanent_uuid: "68366791680f4b03a2706563ed012255" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 38931 } } peers { permanent_uuid: "dd6e2bc7be054d07b458b2942838dd27" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36855 } }
I20260928 06:49:10.569595 26538 raft_consensus.cc:374] T 8fd268d0e6c948ed9c3a1772329385b6 P dd6e2bc7be054d07b458b2942838dd27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:10.569703 26538 raft_consensus.cc:729] T 8fd268d0e6c948ed9c3a1772329385b6 P dd6e2bc7be054d07b458b2942838dd27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd6e2bc7be054d07b458b2942838dd27, State: Initialized, Role: FOLLOWER
I20260928 06:49:10.569887 26539 raft_consensus.cc:348] T 8fd268d0e6c948ed9c3a1772329385b6 P 03878870eb15418a8c1748156b1945f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03878870eb15418a8c1748156b1945f7" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34919 } } peers { permanent_uuid: "68366791680f4b03a2706563ed012255" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 38931 } } peers { permanent_uuid: "dd6e2bc7be054d07b458b2942838dd27" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36855 } }
I20260928 06:49:10.570156 26539 raft_consensus.cc:374] T 8fd268d0e6c948ed9c3a1772329385b6 P 03878870eb15418a8c1748156b1945f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:10.570243 26539 raft_consensus.cc:729] T 8fd268d0e6c948ed9c3a1772329385b6 P 03878870eb15418a8c1748156b1945f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03878870eb15418a8c1748156b1945f7, State: Initialized, Role: FOLLOWER
I20260928 06:49:10.570196 26538 consensus_queue.cc:260] T 8fd268d0e6c948ed9c3a1772329385b6 P dd6e2bc7be054d07b458b2942838dd27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03878870eb15418a8c1748156b1945f7" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34919 } } peers { permanent_uuid: "68366791680f4b03a2706563ed012255" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 38931 } } peers { permanent_uuid: "dd6e2bc7be054d07b458b2942838dd27" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36855 } }
I20260928 06:49:10.570749 26539 consensus_queue.cc:260] T 8fd268d0e6c948ed9c3a1772329385b6 P 03878870eb15418a8c1748156b1945f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03878870eb15418a8c1748156b1945f7" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34919 } } peers { permanent_uuid: "68366791680f4b03a2706563ed012255" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 38931 } } peers { permanent_uuid: "dd6e2bc7be054d07b458b2942838dd27" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36855 } }
I20260928 06:49:10.570784 26537 raft_consensus.cc:348] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03878870eb15418a8c1748156b1945f7" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34919 } } peers { permanent_uuid: "68366791680f4b03a2706563ed012255" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 38931 } } peers { permanent_uuid: "dd6e2bc7be054d07b458b2942838dd27" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36855 } }
I20260928 06:49:10.570964 26389 heartbeater.cc:503] Master 127.7.48.62:37969 was elected leader, sending a full tablet report...
I20260928 06:49:10.571077 26537 raft_consensus.cc:374] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:10.571167 26537 raft_consensus.cc:729] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68366791680f4b03a2706563ed012255, State: Initialized, Role: FOLLOWER
I20260928 06:49:10.571297 26538 ts_tablet_manager.cc:1428] T 8fd268d0e6c948ed9c3a1772329385b6 P dd6e2bc7be054d07b458b2942838dd27: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 06:49:10.571632 26537 consensus_queue.cc:260] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03878870eb15418a8c1748156b1945f7" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34919 } } peers { permanent_uuid: "68366791680f4b03a2706563ed012255" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 38931 } } peers { permanent_uuid: "dd6e2bc7be054d07b458b2942838dd27" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36855 } }
I20260928 06:49:10.571808 26539 ts_tablet_manager.cc:1428] T 8fd268d0e6c948ed9c3a1772329385b6 P 03878870eb15418a8c1748156b1945f7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 06:49:10.572157 26520 heartbeater.cc:503] Master 127.7.48.62:37969 was elected leader, sending a full tablet report...
I20260928 06:49:10.572292 26258 heartbeater.cc:503] Master 127.7.48.62:37969 was elected leader, sending a full tablet report...
I20260928 06:49:10.572553 26537 ts_tablet_manager.cc:1428] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260928 06:49:10.574790 26545 raft_consensus.cc:482] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:49:10.574985 26545 raft_consensus.cc:504] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03878870eb15418a8c1748156b1945f7" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34919 } } peers { permanent_uuid: "68366791680f4b03a2706563ed012255" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 38931 } } peers { permanent_uuid: "dd6e2bc7be054d07b458b2942838dd27" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36855 } }
I20260928 06:49:10.576915 26545 leader_election.cc:288] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03878870eb15418a8c1748156b1945f7 (127.7.48.3:34919), dd6e2bc7be054d07b458b2942838dd27 (127.7.48.2:36855)
I20260928 06:49:10.580546 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:10.576527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:34919 (local address 127.7.48.1:39889)
0928 06:49:10.576772 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:10.576787 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:10.576868 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:10.577344 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:10.577346 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:10.577361 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:10.577451 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:10.577457 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.578399 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.578402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:10.579322 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:10.579331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.580142 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.580147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:10.580245 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:10.580270 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 06:49:10.580347 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 06:49:10.580412 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260928 06:49:10.580538 26534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:10.577025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:39889 (local address 127.7.48.3:34919)
0928 06:49:10.577134 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:10.577138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:10.577150 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:49:10.577191 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:10.577193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:10.577235 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:10.577329 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:10.577599 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.578292 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.579440 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.580031 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.580112 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:10.580155 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:10.580376 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 06:49:10.580426 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:49:10.581003 26536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:10.576681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:53397 (local address 127.7.48.2:36855)
0928 06:49:10.576882 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:10.576886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:10.576994 (+ 108us) server_negotiation.cc:407] Connection header received
0928 06:49:10.577565 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:10.577568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:10.577615 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:10.577698 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:10.578059 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.578739 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.579824 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.580417 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.580553 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:10.580623 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:10.580843 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 06:49:10.580908 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260928 06:49:10.581017 26546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:10.576527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:36855 (local address 127.7.48.1:53397)
0928 06:49:10.576913 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:10.576928 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:10.577419 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:10.577794 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:10.577797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:10.577814 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:10.577922 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:10.577927 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.578851 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.578853 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:10.579700 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:10.579710 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.580594 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.580597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:10.580703 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:10.580732 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 06:49:10.580826 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 06:49:10.580886 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":77,"thread_start_us":102,"threads_started":1}
I20260928 06:49:10.581295 26463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd268d0e6c948ed9c3a1772329385b6" candidate_uuid: "68366791680f4b03a2706563ed012255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03878870eb15418a8c1748156b1945f7" is_pre_election: true
I20260928 06:49:10.581543 26343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd268d0e6c948ed9c3a1772329385b6" candidate_uuid: "68366791680f4b03a2706563ed012255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd6e2bc7be054d07b458b2942838dd27" is_pre_election: true
I20260928 06:49:10.581588 26463 raft_consensus.cc:2454] T 8fd268d0e6c948ed9c3a1772329385b6 P 03878870eb15418a8c1748156b1945f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68366791680f4b03a2706563ed012255 in term 0.
I20260928 06:49:10.581806 26343 raft_consensus.cc:2454] T 8fd268d0e6c948ed9c3a1772329385b6 P dd6e2bc7be054d07b458b2942838dd27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68366791680f4b03a2706563ed012255 in term 0.
I20260928 06:49:10.582062 26147 leader_election.cc:302] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03878870eb15418a8c1748156b1945f7, 68366791680f4b03a2706563ed012255; no voters:
I20260928 06:49:10.582365 26545 raft_consensus.cc:2781] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:49:10.582453 26545 raft_consensus.cc:482] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:49:10.582495 26545 raft_consensus.cc:3037] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:10.583596 26545 raft_consensus.cc:504] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03878870eb15418a8c1748156b1945f7" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34919 } } peers { permanent_uuid: "68366791680f4b03a2706563ed012255" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 38931 } } peers { permanent_uuid: "dd6e2bc7be054d07b458b2942838dd27" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36855 } }
I20260928 06:49:10.584004 26545 leader_election.cc:288] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255 [CANDIDATE]: Term 1 election: Requested vote from peers 03878870eb15418a8c1748156b1945f7 (127.7.48.3:34919), dd6e2bc7be054d07b458b2942838dd27 (127.7.48.2:36855)
I20260928 06:49:10.584273 26463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd268d0e6c948ed9c3a1772329385b6" candidate_uuid: "68366791680f4b03a2706563ed012255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03878870eb15418a8c1748156b1945f7"
I20260928 06:49:10.584357 26343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd268d0e6c948ed9c3a1772329385b6" candidate_uuid: "68366791680f4b03a2706563ed012255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd6e2bc7be054d07b458b2942838dd27"
I20260928 06:49:10.584425 26463 raft_consensus.cc:3037] T 8fd268d0e6c948ed9c3a1772329385b6 P 03878870eb15418a8c1748156b1945f7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:10.584504 26343 raft_consensus.cc:3037] T 8fd268d0e6c948ed9c3a1772329385b6 P dd6e2bc7be054d07b458b2942838dd27 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:10.585438 26463 raft_consensus.cc:2454] T 8fd268d0e6c948ed9c3a1772329385b6 P 03878870eb15418a8c1748156b1945f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68366791680f4b03a2706563ed012255 in term 1.
I20260928 06:49:10.585536 26343 raft_consensus.cc:2454] T 8fd268d0e6c948ed9c3a1772329385b6 P dd6e2bc7be054d07b458b2942838dd27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68366791680f4b03a2706563ed012255 in term 1.
I20260928 06:49:10.585747 26147 leader_election.cc:302] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03878870eb15418a8c1748156b1945f7, 68366791680f4b03a2706563ed012255; no voters:
I20260928 06:49:10.586089 26545 raft_consensus.cc:2781] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:10.586323 26545 raft_consensus.cc:686] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255 [term 1 LEADER]: Becoming Leader. State: Replica: 68366791680f4b03a2706563ed012255, State: Running, Role: LEADER
I20260928 06:49:10.586710 26545 consensus_queue.cc:237] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03878870eb15418a8c1748156b1945f7" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34919 } } peers { permanent_uuid: "68366791680f4b03a2706563ed012255" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 38931 } } peers { permanent_uuid: "dd6e2bc7be054d07b458b2942838dd27" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36855 } }
I20260928 06:49:10.589629 26069 catalog_manager.cc:5495] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68366791680f4b03a2706563ed012255 (127.7.48.1). New cstate: current_term: 1 leader_uuid: "68366791680f4b03a2706563ed012255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03878870eb15418a8c1748156b1945f7" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68366791680f4b03a2706563ed012255" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 38931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd6e2bc7be054d07b458b2942838dd27" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36855 } health_report { overall_health: UNKNOWN } } }
I20260928 06:49:10.605433 26533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:10.601518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39546 (local address 127.7.48.1:38931)
0928 06:49:10.601651 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:10.601654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:10.601668 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:49:10.601902 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:10.601905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:10.601964 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:10.602105 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:49:10.602370 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.602897 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.603858 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:10.604109 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:10.604216 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:10.604579 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:49:10.604977 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:49:10.605141 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:10.605228 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 06:49:10.605265 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 06:49:10.611410 26343 raft_consensus.cc:1261] T 8fd268d0e6c948ed9c3a1772329385b6 P dd6e2bc7be054d07b458b2942838dd27 [term 1 FOLLOWER]: Refusing update from remote peer 68366791680f4b03a2706563ed012255: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:49:10.611450 26463 raft_consensus.cc:1261] T 8fd268d0e6c948ed9c3a1772329385b6 P 03878870eb15418a8c1748156b1945f7 [term 1 FOLLOWER]: Refusing update from remote peer 68366791680f4b03a2706563ed012255: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:49:10.612160 26547 consensus_queue.cc:1035] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd6e2bc7be054d07b458b2942838dd27" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 36855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:49:10.612388 26545 consensus_queue.cc:1035] T 8fd268d0e6c948ed9c3a1772329385b6 P 68366791680f4b03a2706563ed012255 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03878870eb15418a8c1748156b1945f7" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 34919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:49:10.620741 26550 mvcc.cc:205] Tried to move back new op lower bound from 7334208104896503808 to 7334208104805359616. Current Snapshot: MvccSnapshot[applied={T|T < 7334208104896503808}]
I20260928 06:49:10.622882 26552 mvcc.cc:205] Tried to move back new op lower bound from 7334208104896503808 to 7334208104805359616. Current Snapshot: MvccSnapshot[applied={T|T < 7334208104896503808}]
I20260928 06:49:10.629885 26067 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:49:10.630069 26067 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:49:10.632355 26067 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29d80d654967407d8c2ff17833ba2875: Sending DeleteTablet for 3 replicas of tablet 8fd268d0e6c948ed9c3a1772329385b6
I20260928 06:49:10.633137 26454 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fd268d0e6c948ed9c3a1772329385b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:49:10 UTC) from {username='slave'} at 127.0.0.1:50784
I20260928 06:49:10.633229 26323 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fd268d0e6c948ed9c3a1772329385b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:49:10 UTC) from {username='slave'} at 127.0.0.1:52888
I20260928 06:49:10.633262 26192 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fd268d0e6c948ed9c3a1772329385b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:49:10 UTC) from {username='slave'} at 127.0.0.1:39542
I20260928 06:49:10.633692 26560 tablet_replica.cc:330] T 8fd268d0e6c948ed9c3a1772329385b6 P 03878870eb15418a8c1748156b1945f7: stopping tablet replica
I20260928 06:49:10.633839 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 26130
I20260928 06:49:10.634059 26560 raft_consensus.cc:2229] T 8fd268d0e6c948ed9c3a1772329385b6 P 03878870eb15418a8c1748156b1945f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:49:10.634420 26560 raft_consensus.cc:2258] T 8fd268d0e6c948ed9c3a1772329385b6 P 03878870eb15418a8c1748156b1945f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:49:10.635601 26562 tablet_replica.cc:330] T 8fd268d0e6c948ed9c3a1772329385b6 P dd6e2bc7be054d07b458b2942838dd27: stopping tablet replica
I20260928 06:49:10.635952 26562 raft_consensus.cc:2229] T 8fd268d0e6c948ed9c3a1772329385b6 P dd6e2bc7be054d07b458b2942838dd27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:49:10.636292 26562 raft_consensus.cc:2258] T 8fd268d0e6c948ed9c3a1772329385b6 P dd6e2bc7be054d07b458b2942838dd27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:49:10.636927 26560 ts_tablet_manager.cc:1905] T 8fd268d0e6c948ed9c3a1772329385b6 P 03878870eb15418a8c1748156b1945f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:49:10.637763 26562 ts_tablet_manager.cc:1905] T 8fd268d0e6c948ed9c3a1772329385b6 P dd6e2bc7be054d07b458b2942838dd27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:49:10.639171 26560 ts_tablet_manager.cc:1918] T 8fd268d0e6c948ed9c3a1772329385b6 P 03878870eb15418a8c1748156b1945f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:49:10.639282 26560 log.cc:1196] T 8fd268d0e6c948ed9c3a1772329385b6 P 03878870eb15418a8c1748156b1945f7: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/8fd268d0e6c948ed9c3a1772329385b6
I20260928 06:49:10.639616 26560 ts_tablet_manager.cc:1939] T 8fd268d0e6c948ed9c3a1772329385b6 P 03878870eb15418a8c1748156b1945f7: Deleting consensus metadata
I20260928 06:49:10.640834 26055 catalog_manager.cc:4841] TS 03878870eb15418a8c1748156b1945f7 (127.7.48.3:34919): tablet 8fd268d0e6c948ed9c3a1772329385b6 (table test-table [id=9031243c52d346e4a8cd527c81a9e89d]) successfully deleted
I20260928 06:49:10.641194 26562 ts_tablet_manager.cc:1918] T 8fd268d0e6c948ed9c3a1772329385b6 P dd6e2bc7be054d07b458b2942838dd27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:49:10.641299 26562 log.cc:1196] T 8fd268d0e6c948ed9c3a1772329385b6 P dd6e2bc7be054d07b458b2942838dd27: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/8fd268d0e6c948ed9c3a1772329385b6
I20260928 06:49:10.641619 26562 ts_tablet_manager.cc:1939] T 8fd268d0e6c948ed9c3a1772329385b6 P dd6e2bc7be054d07b458b2942838dd27: Deleting consensus metadata
W20260928 06:49:10.642170 26053 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.48.1:38931 (error 108)
I20260928 06:49:10.643005 26053 catalog_manager.cc:4841] TS dd6e2bc7be054d07b458b2942838dd27 (127.7.48.2:36855): tablet 8fd268d0e6c948ed9c3a1772329385b6 (table test-table [id=9031243c52d346e4a8cd527c81a9e89d]) successfully deleted
I20260928 06:49:10.643170 26535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:10.642881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:38931 (local address 127.0.0.1:39558)
0928 06:49:10.642999 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:10.643059 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.1:38931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 06:49:10.643322 26053 catalog_manager.cc:4584] TS 68366791680f4b03a2706563ed012255 (127.7.48.1:38931): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8fd268d0e6c948ed9c3a1772329385b6: Network error: Client connection negotiation failed: client connection to 127.7.48.1:38931: connect: Connection refused (error 111)
I20260928 06:49:10.643493 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 26261
I20260928 06:49:10.649873 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 26392
I20260928 06:49:10.655469 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 26038
2026-09-28T06:49:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (975 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:49:10 dist-test-slave-twl6 krb5kdc[26566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:49:10 dist-test-slave-twl6 krb5kdc[26566](info): set up 2 sockets
Sep 28 06:49:10 dist-test-slave-twl6 krb5kdc[26566](info): commencing operation
krb5kdc: starting...
W20260928 06:49:11.688191 7360 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:49:11 dist-test-slave-twl6 krb5kdc[26566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:49:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:49:11.810247 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:38305
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:38305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:11.920316 26579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:11.920593 26579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:11.920634 26579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:11.924628 26579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:11.924697 26579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:11.924717 26579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:11.924733 26579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:11.924748 26579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:11.930040 26579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:38305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:38305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:11.931145 26579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:11.932193 26579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:11.940809 26585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:11.940888 26584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:11.941005 26579 server_base.cc:1034] running on GCE node
W20260928 06:49:11.940833 26587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:11.941803 26579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:11.942916 26579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:11.944083 26579 hybrid_clock.cc:648] HybridClock initialized: now 1790578151944065 us; error 31 us; skew 500 ppm
Sep 28 06:49:11 dist-test-slave-twl6 krb5kdc[26566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578151, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:11.952229 26579 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:49:11.953298 26579 webserver.cc:460] Webserver started at http://127.7.48.62:42329/ using document root <none> and password file <none>
I20260928 06:49:11.953929 26579 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:11.953994 26579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:11.954241 26579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:11.956481 26579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "0d68ffdc62fd4111b6e48a13cda53076"
format_stamp: "Formatted at 2026-09-28 06:49:11 on dist-test-slave-twl6"
server_key: "404d34928859cc574de39fd0687d0348"
server_key_iv: "1b8a0652df0a81f8064db33ad9572d7b"
server_key_version: "encryptionkey@0"
I20260928 06:49:11.957056 26579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "0d68ffdc62fd4111b6e48a13cda53076"
format_stamp: "Formatted at 2026-09-28 06:49:11 on dist-test-slave-twl6"
server_key: "404d34928859cc574de39fd0687d0348"
server_key_iv: "1b8a0652df0a81f8064db33ad9572d7b"
server_key_version: "encryptionkey@0"
I20260928 06:49:11.960793 26579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:49:11.963335 26594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:11.964581 26579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:49:11.964722 26579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "0d68ffdc62fd4111b6e48a13cda53076"
format_stamp: "Formatted at 2026-09-28 06:49:11 on dist-test-slave-twl6"
server_key: "404d34928859cc574de39fd0687d0348"
server_key_iv: "1b8a0652df0a81f8064db33ad9572d7b"
server_key_version: "encryptionkey@0"
I20260928 06:49:11.964839 26579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:11.984236 26579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:11.987161 26579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:11.987373 26579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:11.994575 26579 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:38305
I20260928 06:49:11.994591 26646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:38305 every 8 connection(s)
I20260928 06:49:11.995525 26579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:11.996822 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 26579
I20260928 06:49:11.996955 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:11.997246 7360 external_mini_cluster.cc:1428] Setting key 6a671eb8a273e67d67c9b5fa42572962
I20260928 06:49:11.998493 26647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:12.004125 26647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d68ffdc62fd4111b6e48a13cda53076: Bootstrap starting.
Sep 28 06:49:12 dist-test-slave-twl6 krb5kdc[26566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:49:12.006628 26647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d68ffdc62fd4111b6e48a13cda53076: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:12.007391 26647 log.cc:824] T 00000000000000000000000000000000 P 0d68ffdc62fd4111b6e48a13cda53076: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:12.009635 26647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d68ffdc62fd4111b6e48a13cda53076: No bootstrap required, opened a new log
I20260928 06:49:12.011925 26650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:11.998647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37014 (local address 127.7.48.62:38305)
0928 06:49:11.999134 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:11.999150 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:11.999186 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:49:11.999822 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:11.999839 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:12.000250 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:12.000593 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:12.001380 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.002421 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.003285 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.003770 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.003986 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:12.006775 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:12.006780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:12.006794 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:12.006823 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:12.008873 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:12.009289 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:12.009294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:12.009300 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:12.009435 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:12.009707 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:12.009710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:12.009712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:12.010198 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:12.010383 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:12.010626 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 06:49:12.010917 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":167,"threads_started":1}
I20260928 06:49:12.014029 26647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d68ffdc62fd4111b6e48a13cda53076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d68ffdc62fd4111b6e48a13cda53076" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 38305 } }
I20260928 06:49:12.014315 26647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d68ffdc62fd4111b6e48a13cda53076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:12.014367 26647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d68ffdc62fd4111b6e48a13cda53076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d68ffdc62fd4111b6e48a13cda53076, State: Initialized, Role: FOLLOWER
I20260928 06:49:12.014732 26647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d68ffdc62fd4111b6e48a13cda53076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d68ffdc62fd4111b6e48a13cda53076" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 38305 } }
I20260928 06:49:12.014838 26647 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d68ffdc62fd4111b6e48a13cda53076 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:49:12.014884 26647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d68ffdc62fd4111b6e48a13cda53076 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:49:12.014951 26647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d68ffdc62fd4111b6e48a13cda53076 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:12.015805 26647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d68ffdc62fd4111b6e48a13cda53076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d68ffdc62fd4111b6e48a13cda53076" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 38305 } }
I20260928 06:49:12.016219 26647 leader_election.cc:302] T 00000000000000000000000000000000 P 0d68ffdc62fd4111b6e48a13cda53076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d68ffdc62fd4111b6e48a13cda53076; no voters:
I20260928 06:49:12.016547 26647 leader_election.cc:288] T 00000000000000000000000000000000 P 0d68ffdc62fd4111b6e48a13cda53076 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:49:12.016696 26652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d68ffdc62fd4111b6e48a13cda53076 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:12.016955 26652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d68ffdc62fd4111b6e48a13cda53076 [term 1 LEADER]: Becoming Leader. State: Replica: 0d68ffdc62fd4111b6e48a13cda53076, State: Running, Role: LEADER
I20260928 06:49:12.017324 26652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d68ffdc62fd4111b6e48a13cda53076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d68ffdc62fd4111b6e48a13cda53076" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 38305 } }
I20260928 06:49:12.017807 26647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d68ffdc62fd4111b6e48a13cda53076 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:12.019299 26654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d68ffdc62fd4111b6e48a13cda53076 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d68ffdc62fd4111b6e48a13cda53076. Latest consensus state: current_term: 1 leader_uuid: "0d68ffdc62fd4111b6e48a13cda53076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d68ffdc62fd4111b6e48a13cda53076" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 38305 } } }
I20260928 06:49:12.019295 26653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d68ffdc62fd4111b6e48a13cda53076 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d68ffdc62fd4111b6e48a13cda53076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d68ffdc62fd4111b6e48a13cda53076" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 38305 } } }
I20260928 06:49:12.019495 26654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d68ffdc62fd4111b6e48a13cda53076 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:12.019529 26653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d68ffdc62fd4111b6e48a13cda53076 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:12.020212 26661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:12.022740 26661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:49:12.028527 26661 catalog_manager.cc:1347] Generated new cluster ID: b293d50690a8495da214fd2934a1c47d
I20260928 06:49:12.028712 26661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:12.062253 26661 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:12.063627 26661 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:12.073915 26661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d68ffdc62fd4111b6e48a13cda53076: Generated new TSK 0
I20260928 06:49:12.074662 26661 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.48.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
Entry for principal kudu/127.7.48.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab.
I20260928 06:49:12.108692 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:38305
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:12.221915 26673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:12.222200 26673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:12.222245 26673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:12.226320 26673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:12.226387 26673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:12.226445 26673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:49:12.231613 26673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.1.keytab
--principal=kudu/127.7.48.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:38305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:12.232764 26673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:12.233796 26673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:12.243712 26678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:12.243703 26679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:12.243703 26681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:12.243911 26673 server_base.cc:1034] running on GCE node
I20260928 06:49:12.244603 26673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:12.245285 26673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:12.246469 26673 hybrid_clock.cc:648] HybridClock initialized: now 1790578152246441 us; error 52 us; skew 500 ppm
Sep 28 06:49:12 dist-test-slave-twl6 krb5kdc[26566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578152, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:12.249265 26673 init.cc:377] Logged in from keytab as kudu/127.7.48.1@KRBTEST.COM (short username kudu)
I20260928 06:49:12.250149 26673 webserver.cc:460] Webserver started at http://127.7.48.1:46473/ using document root <none> and password file <none>
I20260928 06:49:12.250736 26673 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:12.250833 26673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:12.251062 26673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:12.253191 26673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "f902f2a1f1a24c98b0fccf205d84a5a5"
format_stamp: "Formatted at 2026-09-28 06:49:12 on dist-test-slave-twl6"
server_key: "e8cfdb0ecc018d27fd57a25878a609fc"
server_key_iv: "7276d083c36f43d58d71501aeb0d1c1b"
server_key_version: "encryptionkey@0"
I20260928 06:49:12.253787 26673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "f902f2a1f1a24c98b0fccf205d84a5a5"
format_stamp: "Formatted at 2026-09-28 06:49:12 on dist-test-slave-twl6"
server_key: "e8cfdb0ecc018d27fd57a25878a609fc"
server_key_iv: "7276d083c36f43d58d71501aeb0d1c1b"
server_key_version: "encryptionkey@0"
I20260928 06:49:12.258327 26673 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:49:12.261008 26688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:12.262255 26673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:49:12.262358 26673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "f902f2a1f1a24c98b0fccf205d84a5a5"
format_stamp: "Formatted at 2026-09-28 06:49:12 on dist-test-slave-twl6"
server_key: "e8cfdb0ecc018d27fd57a25878a609fc"
server_key_iv: "7276d083c36f43d58d71501aeb0d1c1b"
server_key_version: "encryptionkey@0"
I20260928 06:49:12.262456 26673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:12.278200 26673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:12.281665 26673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:12.281847 26673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:12.282444 26673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:49:12.283253 26673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:49:12.283303 26673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:12.283344 26673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:49:12.283365 26673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:12.292171 26673 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:33555
I20260928 06:49:12.292250 26801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:33555 every 8 connection(s)
I20260928 06:49:12.293162 26673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:49:12.295410 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 26673
I20260928 06:49:12.295553 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
I20260928 06:49:12.295919 7360 external_mini_cluster.cc:1428] Setting key c2e5f124e62ba70dd77d8872528c23d6
Sep 28 06:49:12 dist-test-slave-twl6 krb5kdc[26566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578152, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.1@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.48.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.2@KRBTEST.COM" created.
I20260928 06:49:12.308027 26650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:12.295049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:35335 (local address 127.7.48.62:38305)
0928 06:49:12.295207 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:12.295211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:12.296263 (+ 1052us) server_negotiation.cc:407] Connection header received
0928 06:49:12.297915 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:12.297918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:12.297956 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:12.298014 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:12.299602 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.300217 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.301467 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.301708 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.301820 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:12.304360 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:12.304364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:12.304366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:12.304388 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:12.305839 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:12.306367 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:12.306372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:12.306373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:12.306457 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:12.306804 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:12.306808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:12.306810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:12.306997 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:12.307111 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:12.307670 (+ 559us) server_negotiation.cc:299] Negotiation successful
0928 06:49:12.307809 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 06:49:12.308869 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:12.295480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:38305 (local address 127.7.48.1:35335)
0928 06:49:12.296101 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:12.296155 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:12.297675 (+ 1520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:12.298209 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:12.298221 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:12.298753 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:12.299427 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:12.299441 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.300364 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.300367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:12.301346 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:12.301355 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.301863 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.301867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:12.302046 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:12.302689 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:12.302706 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:12.304185 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:12.305975 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:12.305980 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:12.305997 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:12.306261 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:12.306552 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:12.306555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:12.306557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:12.306708 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:12.307110 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:12.307118 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:12.307385 (+ 267us) client_negotiation.cc:769] Sending connection context
0928 06:49:12.307630 (+ 245us) client_negotiation.cc:240] Negotiation successful
0928 06:49:12.307990 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":151,"threads_started":1}
I20260928 06:49:12.310171 26802 heartbeater.cc:347] Connected to a master server at 127.7.48.62:38305
I20260928 06:49:12.310436 26802 heartbeater.cc:464] Registering TS with master...
I20260928 06:49:12.311015 26802 heartbeater.cc:511] Master 127.7.48.62:38305 requested a full tablet report, sending...
I20260928 06:49:12.312703 26611 ts_manager.cc:194] Registered new tserver with Master: f902f2a1f1a24c98b0fccf205d84a5a5 (127.7.48.1:33555)
I20260928 06:49:12.313863 26611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.1@KRBTEST.COM'} at 127.7.48.1:35335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
Entry for principal kudu/127.7.48.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab.
I20260928 06:49:12.322844 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:38305
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:12.432843 26807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:12.433130 26807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:12.433173 26807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:12.437129 26807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:12.437196 26807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:12.437287 26807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:49:12.442909 26807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.2.keytab
--principal=kudu/127.7.48.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:38305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:12.444015 26807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:12.445061 26807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:12.454873 26813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:12.454888 26815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:12.454873 26812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:12.454910 26807 server_base.cc:1034] running on GCE node
I20260928 06:49:12.455561 26807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:12.456310 26807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:12.457512 26807 hybrid_clock.cc:648] HybridClock initialized: now 1790578152457489 us; error 38 us; skew 500 ppm
Sep 28 06:49:12 dist-test-slave-twl6 krb5kdc[26566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578152, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:12.460454 26807 init.cc:377] Logged in from keytab as kudu/127.7.48.2@KRBTEST.COM (short username kudu)
I20260928 06:49:12.461340 26807 webserver.cc:460] Webserver started at http://127.7.48.2:39117/ using document root <none> and password file <none>
I20260928 06:49:12.461915 26807 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:12.461977 26807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:12.462219 26807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:12.464354 26807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "ef7d7fea346c4cb3a1cc77396448cf2d"
format_stamp: "Formatted at 2026-09-28 06:49:12 on dist-test-slave-twl6"
server_key: "f10fbca1aff4f39b2ef40a19ea63ad6d"
server_key_iv: "1d05087bbd7628c4ac2cb4cc2db16481"
server_key_version: "encryptionkey@0"
I20260928 06:49:12.464926 26807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "ef7d7fea346c4cb3a1cc77396448cf2d"
format_stamp: "Formatted at 2026-09-28 06:49:12 on dist-test-slave-twl6"
server_key: "f10fbca1aff4f39b2ef40a19ea63ad6d"
server_key_iv: "1d05087bbd7628c4ac2cb4cc2db16481"
server_key_version: "encryptionkey@0"
I20260928 06:49:12.468923 26807 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 06:49:12.471361 26822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:12.472723 26807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:49:12.472849 26807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "ef7d7fea346c4cb3a1cc77396448cf2d"
format_stamp: "Formatted at 2026-09-28 06:49:12 on dist-test-slave-twl6"
server_key: "f10fbca1aff4f39b2ef40a19ea63ad6d"
server_key_iv: "1d05087bbd7628c4ac2cb4cc2db16481"
server_key_version: "encryptionkey@0"
I20260928 06:49:12.472958 26807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:12.501904 26807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:12.505409 26807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:12.505584 26807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:12.506240 26807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:49:12.507108 26807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:49:12.507155 26807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:12.507227 26807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:49:12.507261 26807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:12.516691 26807 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:33699
I20260928 06:49:12.516706 26935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:33699 every 8 connection(s)
I20260928 06:49:12.517992 26807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:49:12.519901 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 26807
I20260928 06:49:12.520028 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:49:12.520391 7360 external_mini_cluster.cc:1428] Setting key db25968b85ded9b104de2033c0498747
Sep 28 06:49:12 dist-test-slave-twl6 krb5kdc[26566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578152, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.2@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.48.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.3@KRBTEST.COM" created.
I20260928 06:49:12.535475 26650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:12.521146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:54541 (local address 127.7.48.62:38305)
0928 06:49:12.521446 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:12.521451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:12.522547 (+ 1096us) server_negotiation.cc:407] Connection header received
0928 06:49:12.524838 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:12.524843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:12.524903 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:12.524990 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:12.526710 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.527373 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.528692 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.528919 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.529032 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:12.531792 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:12.531795 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:12.531797 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:12.531823 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:12.533333 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:12.533831 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:12.533856 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:12.533860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:12.533954 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:12.534342 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:12.534344 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:12.534345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:12.534530 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:12.534623 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:12.535253 (+ 630us) server_negotiation.cc:299] Negotiation successful
0928 06:49:12.535352 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
I20260928 06:49:12.536893 26938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:12.521367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:38305 (local address 127.7.48.2:54541)
0928 06:49:12.522365 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:12.522445 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:12.524402 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:12.525276 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:12.525286 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:12.525811 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:12.526521 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:12.526545 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.527540 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.527543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:12.528558 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:12.528568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.529101 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.529105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:12.529276 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:12.529932 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:12.529949 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:12.531604 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:12.533447 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:12.533452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:12.533466 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:12.533721 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:12.534060 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:12.534064 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:12.534069 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:12.534220 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:12.534647 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:12.534656 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:12.535000 (+ 344us) client_negotiation.cc:769] Sending connection context
0928 06:49:12.535220 (+ 220us) client_negotiation.cc:240] Negotiation successful
0928 06:49:12.535624 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":160,"threads_started":1}
I20260928 06:49:12.538182 26936 heartbeater.cc:347] Connected to a master server at 127.7.48.62:38305
I20260928 06:49:12.538450 26936 heartbeater.cc:464] Registering TS with master...
I20260928 06:49:12.539093 26936 heartbeater.cc:511] Master 127.7.48.62:38305 requested a full tablet report, sending...
I20260928 06:49:12.540323 26611 ts_manager.cc:194] Registered new tserver with Master: ef7d7fea346c4cb3a1cc77396448cf2d (127.7.48.2:33699)
I20260928 06:49:12.540900 26611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.2@KRBTEST.COM'} at 127.7.48.2:54541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
Entry for principal kudu/127.7.48.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab.
I20260928 06:49:12.550776 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:38305
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:12.660621 26941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:12.660892 26941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:12.660969 26941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:12.665166 26941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:12.665235 26941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:12.665328 26941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:49:12.670845 26941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.3.keytab
--principal=kudu/127.7.48.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:38305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:12.672004 26941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:12.673076 26941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:12.684115 26947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:12.684115 26946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:12.684129 26949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:12.684718 26941 server_base.cc:1034] running on GCE node
I20260928 06:49:12.685078 26941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:12.685663 26941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:12.686842 26941 hybrid_clock.cc:648] HybridClock initialized: now 1790578152686819 us; error 35 us; skew 500 ppm
Sep 28 06:49:12 dist-test-slave-twl6 krb5kdc[26566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578152, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:12.689709 26941 init.cc:377] Logged in from keytab as kudu/127.7.48.3@KRBTEST.COM (short username kudu)
I20260928 06:49:12.690656 26941 webserver.cc:460] Webserver started at http://127.7.48.3:39791/ using document root <none> and password file <none>
I20260928 06:49:12.691226 26941 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:12.691318 26941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:12.691550 26941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:12.693728 26941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "7077b23fb5204aa69b2bd37a53273fbc"
format_stamp: "Formatted at 2026-09-28 06:49:12 on dist-test-slave-twl6"
server_key: "e4d6e19d2de79fe59a56a658778702ba"
server_key_iv: "81822591b43e5a79fd4cfec4ddf80ea2"
server_key_version: "encryptionkey@0"
I20260928 06:49:12.694300 26941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "7077b23fb5204aa69b2bd37a53273fbc"
format_stamp: "Formatted at 2026-09-28 06:49:12 on dist-test-slave-twl6"
server_key: "e4d6e19d2de79fe59a56a658778702ba"
server_key_iv: "81822591b43e5a79fd4cfec4ddf80ea2"
server_key_version: "encryptionkey@0"
I20260928 06:49:12.698069 26941 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 06:49:12.700551 26956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:12.701705 26941 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:49:12.701841 26941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "7077b23fb5204aa69b2bd37a53273fbc"
format_stamp: "Formatted at 2026-09-28 06:49:12 on dist-test-slave-twl6"
server_key: "e4d6e19d2de79fe59a56a658778702ba"
server_key_iv: "81822591b43e5a79fd4cfec4ddf80ea2"
server_key_version: "encryptionkey@0"
I20260928 06:49:12.701952 26941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:12.722337 26941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:12.725926 26941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:12.726147 26941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:12.726828 26941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:49:12.727691 26941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:49:12.727762 26941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:12.727831 26941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:49:12.727907 26941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:12.736675 26941 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:37661
I20260928 06:49:12.736699 27069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:37661 every 8 connection(s)
I20260928 06:49:12.737602 26941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:49:12.741014 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 26941
I20260928 06:49:12.741178 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:49:12.741462 7360 external_mini_cluster.cc:1428] Setting key cefccbb707cdb5cfb07c8c725dad2890
Sep 28 06:49:12 dist-test-slave-twl6 krb5kdc[26566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578152, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.3@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:49:12.751353 26650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:12.739539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:37249 (local address 127.7.48.62:38305)
0928 06:49:12.739698 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:12.739702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:12.740617 (+ 915us) server_negotiation.cc:407] Connection header received
0928 06:49:12.742055 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:12.742058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:12.742113 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:12.742219 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:12.743623 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.744249 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.745398 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.745604 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.745678 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:12.748085 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:12.748087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:12.748090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:12.748113 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:12.749579 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:12.750032 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:12.750034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:12.750035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:12.750106 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:12.750393 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:12.750395 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:12.750395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:12.750563 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:12.750635 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:12.751140 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 06:49:12.751234 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:49:12.752331 27072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:12.739924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:38305 (local address 127.7.48.3:37249)
0928 06:49:12.740446 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:12.740512 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:12.741848 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:12.742398 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:12.742409 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:12.742912 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:12.743460 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:12.743481 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.744371 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.744374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:12.745290 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:12.745300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.745732 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.745735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:12.745872 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:12.746424 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:12.746461 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:12.747855 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:12.749680 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:12.749684 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:12.749695 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:12.749949 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:12.750185 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:12.750187 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:12.750188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:12.750316 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:12.750678 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:12.750683 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:12.750922 (+ 239us) client_negotiation.cc:769] Sending connection context
0928 06:49:12.751100 (+ 178us) client_negotiation.cc:240] Negotiation successful
0928 06:49:12.751359 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":172,"threads_started":1}
I20260928 06:49:12.753600 27070 heartbeater.cc:347] Connected to a master server at 127.7.48.62:38305
I20260928 06:49:12.753871 27070 heartbeater.cc:464] Registering TS with master...
I20260928 06:49:12.754449 27070 heartbeater.cc:511] Master 127.7.48.62:38305 requested a full tablet report, sending...
I20260928 06:49:12.755470 26611 ts_manager.cc:194] Registered new tserver with Master: 7077b23fb5204aa69b2bd37a53273fbc (127.7.48.3:37661)
I20260928 06:49:12.756055 26611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.48.3@KRBTEST.COM'} at 127.7.48.3:37249
I20260928 06:49:12.767341 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:49:12.776276 26650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:12.769417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37028 (local address 127.7.48.62:38305)
0928 06:49:12.769558 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:12.769563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:12.769728 (+ 165us) server_negotiation.cc:407] Connection header received
0928 06:49:12.769816 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:12.769818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:12.769858 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:12.769920 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:12.770601 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.771094 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.771871 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.772099 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.772171 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:12.773481 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:12.773484 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:12.773486 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:12.773508 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:12.774772 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:12.775148 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:12.775150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:12.775151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:12.775222 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:12.775437 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:12.775439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:12.775440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:12.775639 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:12.775738 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:12.775997 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 06:49:12.776128 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:49:12.779299 26611 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:49:12.783083 26611 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:49:12.798070 27083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:12.792418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52950 (local address 127.7.48.2:33699)
0928 06:49:12.793081 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:12.793095 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:12.793120 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:49:12.793200 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:12.793207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:12.793448 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:12.793626 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:12.793923 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.795074 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.796714 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.797383 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.797524 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:12.797671 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:12.797806 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 06:49:12.797895 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":168,"threads_started":1}
I20260928 06:49:12.798065 27081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:12.792253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:33699 (local address 127.0.0.1:52950)
0928 06:49:12.792720 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:12.792767 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:12.792894 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:12.793612 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:12.793618 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:12.793643 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:12.793800 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:12.793809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.795200 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.795203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:12.796590 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:12.796600 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.797524 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.797527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:12.797629 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:12.797664 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 06:49:12.797778 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 06:49:12.797895 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":98,"threads_started":1}
I20260928 06:49:12.798306 27085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:12.792346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:33555 (local address 127.0.0.1:41480)
0928 06:49:12.793922 (+ 1576us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:12.793937 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:12.794028 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:12.794531 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:12.794534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:12.794552 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:12.794636 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:12.794642 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.795719 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.795723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:12.796681 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:12.796691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.797911 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.797914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:12.798014 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:12.798035 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 06:49:12.798116 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 06:49:12.798181 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1488,"thread_start_us":109,"threads_started":1}
I20260928 06:49:12.798782 27084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:12.793121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41480 (local address 127.7.48.1:33555)
0928 06:49:12.793453 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:12.793477 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:12.794117 (+ 640us) server_negotiation.cc:407] Connection header received
0928 06:49:12.794197 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:12.794201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:12.794389 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:12.794534 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:12.794729 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.795593 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.796821 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.797632 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.797723 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:12.797851 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:12.798512 (+ 661us) server_negotiation.cc:299] Negotiation successful
0928 06:49:12.798626 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":103,"threads_started":1}
I20260928 06:49:12.798851 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:12.793477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:37661 (local address 127.0.0.1:50352)
0928 06:49:12.793980 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:12.793993 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:12.794075 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:12.794866 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:12.794868 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:12.794884 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:12.794986 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:12.794992 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.796344 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.796347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:12.797391 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:12.797403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.798465 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.798471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:12.798595 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:12.798617 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 06:49:12.798668 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:49:12.798731 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":93,"threads_started":1}
I20260928 06:49:12.799501 27086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:12.794009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50352 (local address 127.7.48.3:37661)
0928 06:49:12.794429 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:12.794438 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:12.794456 (+ 18us) server_negotiation.cc:407] Connection header received
0928 06:49:12.794545 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:12.794550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:12.794732 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:12.794888 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:12.795076 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.796209 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.797535 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:12.798309 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:12.799099 (+ 790us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:12.799230 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:12.799275 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 06:49:12.799350 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":141,"threads_started":1}
I20260928 06:49:12.801703 27004 tablet_service.cc:1467] Processing CreateTablet for tablet 938575a9c1ba4e38b9a39e727db39f93 (DEFAULT_TABLE table=test-table [id=b3d1d1b696c24fa9b62f5c5a4e48759f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:49:12.801620 26870 tablet_service.cc:1467] Processing CreateTablet for tablet 938575a9c1ba4e38b9a39e727db39f93 (DEFAULT_TABLE table=test-table [id=b3d1d1b696c24fa9b62f5c5a4e48759f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:49:12.801621 26736 tablet_service.cc:1467] Processing CreateTablet for tablet 938575a9c1ba4e38b9a39e727db39f93 (DEFAULT_TABLE table=test-table [id=b3d1d1b696c24fa9b62f5c5a4e48759f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:49:12.802644 27004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938575a9c1ba4e38b9a39e727db39f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:12.802664 26870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938575a9c1ba4e38b9a39e727db39f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:12.802664 26736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938575a9c1ba4e38b9a39e727db39f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:12.808929 27087 tablet_bootstrap.cc:492] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc: Bootstrap starting.
I20260928 06:49:12.810135 27088 tablet_bootstrap.cc:492] T 938575a9c1ba4e38b9a39e727db39f93 P ef7d7fea346c4cb3a1cc77396448cf2d: Bootstrap starting.
I20260928 06:49:12.811311 27087 tablet_bootstrap.cc:654] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:12.811753 27088 tablet_bootstrap.cc:654] T 938575a9c1ba4e38b9a39e727db39f93 P ef7d7fea346c4cb3a1cc77396448cf2d: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:12.812225 27089 tablet_bootstrap.cc:492] T 938575a9c1ba4e38b9a39e727db39f93 P f902f2a1f1a24c98b0fccf205d84a5a5: Bootstrap starting.
I20260928 06:49:12.812433 27087 log.cc:824] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:12.812984 27088 log.cc:824] T 938575a9c1ba4e38b9a39e727db39f93 P ef7d7fea346c4cb3a1cc77396448cf2d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:12.814324 27089 tablet_bootstrap.cc:654] T 938575a9c1ba4e38b9a39e727db39f93 P f902f2a1f1a24c98b0fccf205d84a5a5: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:12.814872 27088 tablet_bootstrap.cc:492] T 938575a9c1ba4e38b9a39e727db39f93 P ef7d7fea346c4cb3a1cc77396448cf2d: No bootstrap required, opened a new log
I20260928 06:49:12.814936 27087 tablet_bootstrap.cc:492] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc: No bootstrap required, opened a new log
I20260928 06:49:12.815081 27088 ts_tablet_manager.cc:1397] T 938575a9c1ba4e38b9a39e727db39f93 P ef7d7fea346c4cb3a1cc77396448cf2d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 06:49:12.815114 27087 ts_tablet_manager.cc:1397] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260928 06:49:12.815102 27089 log.cc:824] T 938575a9c1ba4e38b9a39e727db39f93 P f902f2a1f1a24c98b0fccf205d84a5a5: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:12.816941 27089 tablet_bootstrap.cc:492] T 938575a9c1ba4e38b9a39e727db39f93 P f902f2a1f1a24c98b0fccf205d84a5a5: No bootstrap required, opened a new log
I20260928 06:49:12.817126 27089 ts_tablet_manager.cc:1397] T 938575a9c1ba4e38b9a39e727db39f93 P f902f2a1f1a24c98b0fccf205d84a5a5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 06:49:12.818090 27088 raft_consensus.cc:348] T 938575a9c1ba4e38b9a39e727db39f93 P ef7d7fea346c4cb3a1cc77396448cf2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef7d7fea346c4cb3a1cc77396448cf2d" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33699 } } peers { permanent_uuid: "f902f2a1f1a24c98b0fccf205d84a5a5" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 33555 } } peers { permanent_uuid: "7077b23fb5204aa69b2bd37a53273fbc" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 37661 } }
I20260928 06:49:12.818501 27088 raft_consensus.cc:374] T 938575a9c1ba4e38b9a39e727db39f93 P ef7d7fea346c4cb3a1cc77396448cf2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:12.818607 27088 raft_consensus.cc:729] T 938575a9c1ba4e38b9a39e727db39f93 P ef7d7fea346c4cb3a1cc77396448cf2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef7d7fea346c4cb3a1cc77396448cf2d, State: Initialized, Role: FOLLOWER
I20260928 06:49:12.819020 27087 raft_consensus.cc:348] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef7d7fea346c4cb3a1cc77396448cf2d" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33699 } } peers { permanent_uuid: "f902f2a1f1a24c98b0fccf205d84a5a5" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 33555 } } peers { permanent_uuid: "7077b23fb5204aa69b2bd37a53273fbc" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 37661 } }
I20260928 06:49:12.819111 27088 consensus_queue.cc:260] T 938575a9c1ba4e38b9a39e727db39f93 P ef7d7fea346c4cb3a1cc77396448cf2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef7d7fea346c4cb3a1cc77396448cf2d" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33699 } } peers { permanent_uuid: "f902f2a1f1a24c98b0fccf205d84a5a5" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 33555 } } peers { permanent_uuid: "7077b23fb5204aa69b2bd37a53273fbc" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 37661 } }
I20260928 06:49:12.819383 27087 raft_consensus.cc:374] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:12.819489 27087 raft_consensus.cc:729] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7077b23fb5204aa69b2bd37a53273fbc, State: Initialized, Role: FOLLOWER
I20260928 06:49:12.819769 26936 heartbeater.cc:503] Master 127.7.48.62:38305 was elected leader, sending a full tablet report...
I20260928 06:49:12.820104 27088 ts_tablet_manager.cc:1428] T 938575a9c1ba4e38b9a39e727db39f93 P ef7d7fea346c4cb3a1cc77396448cf2d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 06:49:12.820562 27087 consensus_queue.cc:260] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef7d7fea346c4cb3a1cc77396448cf2d" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33699 } } peers { permanent_uuid: "f902f2a1f1a24c98b0fccf205d84a5a5" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 33555 } } peers { permanent_uuid: "7077b23fb5204aa69b2bd37a53273fbc" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 37661 } }
I20260928 06:49:12.821197 27070 heartbeater.cc:503] Master 127.7.48.62:38305 was elected leader, sending a full tablet report...
I20260928 06:49:12.821412 27087 ts_tablet_manager.cc:1428] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260928 06:49:12.822132 27089 raft_consensus.cc:348] T 938575a9c1ba4e38b9a39e727db39f93 P f902f2a1f1a24c98b0fccf205d84a5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef7d7fea346c4cb3a1cc77396448cf2d" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33699 } } peers { permanent_uuid: "f902f2a1f1a24c98b0fccf205d84a5a5" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 33555 } } peers { permanent_uuid: "7077b23fb5204aa69b2bd37a53273fbc" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 37661 } }
I20260928 06:49:12.822371 27089 raft_consensus.cc:374] T 938575a9c1ba4e38b9a39e727db39f93 P f902f2a1f1a24c98b0fccf205d84a5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:12.822474 27089 raft_consensus.cc:729] T 938575a9c1ba4e38b9a39e727db39f93 P f902f2a1f1a24c98b0fccf205d84a5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f902f2a1f1a24c98b0fccf205d84a5a5, State: Initialized, Role: FOLLOWER
I20260928 06:49:12.822932 27089 consensus_queue.cc:260] T 938575a9c1ba4e38b9a39e727db39f93 P f902f2a1f1a24c98b0fccf205d84a5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef7d7fea346c4cb3a1cc77396448cf2d" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33699 } } peers { permanent_uuid: "f902f2a1f1a24c98b0fccf205d84a5a5" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 33555 } } peers { permanent_uuid: "7077b23fb5204aa69b2bd37a53273fbc" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 37661 } }
I20260928 06:49:12.823691 26802 heartbeater.cc:503] Master 127.7.48.62:38305 was elected leader, sending a full tablet report...
I20260928 06:49:12.823832 27089 ts_tablet_manager.cc:1428] T 938575a9c1ba4e38b9a39e727db39f93 P f902f2a1f1a24c98b0fccf205d84a5a5: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
W20260928 06:49:12.988884 27071 tablet.cc:2369] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:49:13.008205 27094 raft_consensus.cc:482] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:49:13.008412 27094 raft_consensus.cc:504] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef7d7fea346c4cb3a1cc77396448cf2d" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33699 } } peers { permanent_uuid: "f902f2a1f1a24c98b0fccf205d84a5a5" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 33555 } } peers { permanent_uuid: "7077b23fb5204aa69b2bd37a53273fbc" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 37661 } }
I20260928 06:49:13.009491 27094 leader_election.cc:288] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef7d7fea346c4cb3a1cc77396448cf2d (127.7.48.2:33699), f902f2a1f1a24c98b0fccf205d84a5a5 (127.7.48.1:33555)
I20260928 06:49:13.014434 27084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:13.009892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:40603 (local address 127.7.48.1:33555)
0928 06:49:13.010019 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:13.010023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:13.010039 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:49:13.010196 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:13.010202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:13.010263 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:13.010369 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:13.010682 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.011408 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.013082 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.013698 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.013822 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:13.013885 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:13.014231 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 06:49:13.014305 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 06:49:13.014436 27083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:13.010056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:58523 (local address 127.7.48.2:33699)
0928 06:49:13.010188 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:13.010192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:13.010325 (+ 133us) server_negotiation.cc:407] Connection header received
0928 06:49:13.010420 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:13.010423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:13.010486 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:13.010547 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:13.010808 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.011460 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.013084 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.013698 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.013822 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:13.013885 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:13.014231 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 06:49:13.014305 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:49:13.014470 27072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:13.009792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:33555 (local address 127.7.48.3:40603)
0928 06:49:13.009923 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:13.009939 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:13.010057 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:13.010402 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:13.010406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:13.010425 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:13.010569 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:13.010575 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.011534 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.011537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:13.012959 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:13.012966 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.013844 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.013848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:13.014058 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:13.014107 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 06:49:13.014188 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 06:49:13.014307 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":138}
I20260928 06:49:13.014516 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:13.009801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:33699 (local address 127.7.48.3:58523)
0928 06:49:13.010231 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:13.010242 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:13.010318 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:13.010580 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:13.010582 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:13.010597 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:13.010709 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:13.010714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.011552 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.011554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:13.012949 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:13.012962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.013844 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.013848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:13.014028 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:13.014107 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 06:49:13.014188 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 06:49:13.014307 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":258,"thread_start_us":124,"threads_started":1}
I20260928 06:49:13.015273 26890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938575a9c1ba4e38b9a39e727db39f93" candidate_uuid: "7077b23fb5204aa69b2bd37a53273fbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef7d7fea346c4cb3a1cc77396448cf2d" is_pre_election: true
I20260928 06:49:13.015344 26756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938575a9c1ba4e38b9a39e727db39f93" candidate_uuid: "7077b23fb5204aa69b2bd37a53273fbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f902f2a1f1a24c98b0fccf205d84a5a5" is_pre_election: true
I20260928 06:49:13.015654 26756 raft_consensus.cc:2454] T 938575a9c1ba4e38b9a39e727db39f93 P f902f2a1f1a24c98b0fccf205d84a5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7077b23fb5204aa69b2bd37a53273fbc in term 0.
I20260928 06:49:13.015654 26890 raft_consensus.cc:2454] T 938575a9c1ba4e38b9a39e727db39f93 P ef7d7fea346c4cb3a1cc77396448cf2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7077b23fb5204aa69b2bd37a53273fbc in term 0.
I20260928 06:49:13.016252 26957 leader_election.cc:302] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7077b23fb5204aa69b2bd37a53273fbc, ef7d7fea346c4cb3a1cc77396448cf2d, f902f2a1f1a24c98b0fccf205d84a5a5; no voters:
I20260928 06:49:13.016580 27094 raft_consensus.cc:2781] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:49:13.016685 27094 raft_consensus.cc:482] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:49:13.016783 27094 raft_consensus.cc:3037] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:13.017906 27094 raft_consensus.cc:504] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef7d7fea346c4cb3a1cc77396448cf2d" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33699 } } peers { permanent_uuid: "f902f2a1f1a24c98b0fccf205d84a5a5" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 33555 } } peers { permanent_uuid: "7077b23fb5204aa69b2bd37a53273fbc" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 37661 } }
I20260928 06:49:13.018319 27094 leader_election.cc:288] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc [CANDIDATE]: Term 1 election: Requested vote from peers ef7d7fea346c4cb3a1cc77396448cf2d (127.7.48.2:33699), f902f2a1f1a24c98b0fccf205d84a5a5 (127.7.48.1:33555)
I20260928 06:49:13.018707 26756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938575a9c1ba4e38b9a39e727db39f93" candidate_uuid: "7077b23fb5204aa69b2bd37a53273fbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f902f2a1f1a24c98b0fccf205d84a5a5"
I20260928 06:49:13.018814 26890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938575a9c1ba4e38b9a39e727db39f93" candidate_uuid: "7077b23fb5204aa69b2bd37a53273fbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef7d7fea346c4cb3a1cc77396448cf2d"
I20260928 06:49:13.018831 26756 raft_consensus.cc:3037] T 938575a9c1ba4e38b9a39e727db39f93 P f902f2a1f1a24c98b0fccf205d84a5a5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:13.018977 26890 raft_consensus.cc:3037] T 938575a9c1ba4e38b9a39e727db39f93 P ef7d7fea346c4cb3a1cc77396448cf2d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:13.020076 26756 raft_consensus.cc:2454] T 938575a9c1ba4e38b9a39e727db39f93 P f902f2a1f1a24c98b0fccf205d84a5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7077b23fb5204aa69b2bd37a53273fbc in term 1.
I20260928 06:49:13.020084 26890 raft_consensus.cc:2454] T 938575a9c1ba4e38b9a39e727db39f93 P ef7d7fea346c4cb3a1cc77396448cf2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7077b23fb5204aa69b2bd37a53273fbc in term 1.
I20260928 06:49:13.020396 26959 leader_election.cc:302] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7077b23fb5204aa69b2bd37a53273fbc, f902f2a1f1a24c98b0fccf205d84a5a5; no voters:
I20260928 06:49:13.020601 27094 raft_consensus.cc:2781] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:13.020879 27094 raft_consensus.cc:686] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc [term 1 LEADER]: Becoming Leader. State: Replica: 7077b23fb5204aa69b2bd37a53273fbc, State: Running, Role: LEADER
I20260928 06:49:13.021229 27094 consensus_queue.cc:237] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef7d7fea346c4cb3a1cc77396448cf2d" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33699 } } peers { permanent_uuid: "f902f2a1f1a24c98b0fccf205d84a5a5" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 33555 } } peers { permanent_uuid: "7077b23fb5204aa69b2bd37a53273fbc" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 37661 } }
W20260928 06:49:13.022280 26937 tablet.cc:2369] T 938575a9c1ba4e38b9a39e727db39f93 P ef7d7fea346c4cb3a1cc77396448cf2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:49:13.024504 26610 catalog_manager.cc:5495] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc reported cstate change: term changed from 0 to 1, leader changed from <none> to 7077b23fb5204aa69b2bd37a53273fbc (127.7.48.3). New cstate: current_term: 1 leader_uuid: "7077b23fb5204aa69b2bd37a53273fbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef7d7fea346c4cb3a1cc77396448cf2d" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f902f2a1f1a24c98b0fccf205d84a5a5" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 33555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7077b23fb5204aa69b2bd37a53273fbc" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 37661 } health_report { overall_health: HEALTHY } } }
I20260928 06:49:13.036274 27086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:13.032995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50360 (local address 127.7.48.3:37661)
0928 06:49:13.033113 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:13.033117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:13.033129 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:49:13.033246 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:13.033249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:13.033300 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:13.033388 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:49:13.033633 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.034115 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.034923 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.035124 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.035197 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:13.035446 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:49:13.035849 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:49:13.036015 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:13.036071 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 06:49:13.036139 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:49:13.042066 26756 raft_consensus.cc:1261] T 938575a9c1ba4e38b9a39e727db39f93 P f902f2a1f1a24c98b0fccf205d84a5a5 [term 1 FOLLOWER]: Refusing update from remote peer 7077b23fb5204aa69b2bd37a53273fbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:49:13.042073 26890 raft_consensus.cc:1261] T 938575a9c1ba4e38b9a39e727db39f93 P ef7d7fea346c4cb3a1cc77396448cf2d [term 1 FOLLOWER]: Refusing update from remote peer 7077b23fb5204aa69b2bd37a53273fbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:49:13.042950 27097 consensus_queue.cc:1035] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f902f2a1f1a24c98b0fccf205d84a5a5" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:49:13.043164 27094 consensus_queue.cc:1035] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef7d7fea346c4cb3a1cc77396448cf2d" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 06:49:13.044500 26803 tablet.cc:2369] T 938575a9c1ba4e38b9a39e727db39f93 P f902f2a1f1a24c98b0fccf205d84a5a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:49:13.050531 27100 mvcc.cc:205] Tried to move back new op lower bound from 7334208114852581376 to 7334208114777456640. Current Snapshot: MvccSnapshot[applied={T|T < 7334208114852581376}]
I20260928 06:49:13.053359 27101 mvcc.cc:205] Tried to move back new op lower bound from 7334208114852581376 to 7334208114777456640. Current Snapshot: MvccSnapshot[applied={T|T < 7334208114852581376}]
I20260928 06:49:13.059533 26610 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:49:13.059736 26610 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:49:13.062209 26610 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d68ffdc62fd4111b6e48a13cda53076: Sending DeleteTablet for 3 replicas of tablet 938575a9c1ba4e38b9a39e727db39f93
I20260928 06:49:13.063002 26870 tablet_service.cc:1514] Processing DeleteTablet for tablet 938575a9c1ba4e38b9a39e727db39f93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:49:13 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:52950
I20260928 06:49:13.063005 26736 tablet_service.cc:1514] Processing DeleteTablet for tablet 938575a9c1ba4e38b9a39e727db39f93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:49:13 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:41480
I20260928 06:49:13.063035 27004 tablet_service.cc:1514] Processing DeleteTablet for tablet 938575a9c1ba4e38b9a39e727db39f93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:49:13 UTC) from {username='kudu', principal='kudu/127.7.48.62@KRBTEST.COM'} at 127.0.0.1:50352
I20260928 06:49:13.063521 27111 tablet_replica.cc:330] T 938575a9c1ba4e38b9a39e727db39f93 P ef7d7fea346c4cb3a1cc77396448cf2d: stopping tablet replica
I20260928 06:49:13.063793 27109 tablet_replica.cc:330] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc: stopping tablet replica
I20260928 06:49:13.064126 27109 raft_consensus.cc:2229] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:49:13.064294 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 26673
I20260928 06:49:13.065343 27109 raft_consensus.cc:2258] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:49:13.066602 27111 raft_consensus.cc:2229] T 938575a9c1ba4e38b9a39e727db39f93 P ef7d7fea346c4cb3a1cc77396448cf2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:49:13.066928 27111 raft_consensus.cc:2258] T 938575a9c1ba4e38b9a39e727db39f93 P ef7d7fea346c4cb3a1cc77396448cf2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:49:13.067350 27109 ts_tablet_manager.cc:1905] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:49:13.068552 27111 ts_tablet_manager.cc:1905] T 938575a9c1ba4e38b9a39e727db39f93 P ef7d7fea346c4cb3a1cc77396448cf2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:49:13.070348 27109 ts_tablet_manager.cc:1918] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:49:13.070518 27109 log.cc:1196] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/938575a9c1ba4e38b9a39e727db39f93
I20260928 06:49:13.070919 27109 ts_tablet_manager.cc:1939] T 938575a9c1ba4e38b9a39e727db39f93 P 7077b23fb5204aa69b2bd37a53273fbc: Deleting consensus metadata
I20260928 06:49:13.071517 27111 ts_tablet_manager.cc:1918] T 938575a9c1ba4e38b9a39e727db39f93 P ef7d7fea346c4cb3a1cc77396448cf2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:49:13.071676 27111 log.cc:1196] T 938575a9c1ba4e38b9a39e727db39f93 P ef7d7fea346c4cb3a1cc77396448cf2d: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577905526996-7360-0/minicluster-data/ts-1/wal/wals/938575a9c1ba4e38b9a39e727db39f93
I20260928 06:49:13.072065 27111 ts_tablet_manager.cc:1939] T 938575a9c1ba4e38b9a39e727db39f93 P ef7d7fea346c4cb3a1cc77396448cf2d: Deleting consensus metadata
I20260928 06:49:13.072393 26596 catalog_manager.cc:4841] TS 7077b23fb5204aa69b2bd37a53273fbc (127.7.48.3:37661): tablet 938575a9c1ba4e38b9a39e727db39f93 (table test-table [id=b3d1d1b696c24fa9b62f5c5a4e48759f]) successfully deleted
W20260928 06:49:13.073145 26597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.48.1:33555 (error 108)
I20260928 06:49:13.073417 26595 catalog_manager.cc:4841] TS ef7d7fea346c4cb3a1cc77396448cf2d (127.7.48.2:33699): tablet 938575a9c1ba4e38b9a39e727db39f93 (table test-table [id=b3d1d1b696c24fa9b62f5c5a4e48759f]) successfully deleted
I20260928 06:49:13.074138 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:13.073875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:33555 (local address 127.0.0.1:41486)
0928 06:49:13.074001 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:13.074068 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.1:33555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 06:49:13.074318 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 26807
W20260928 06:49:13.074316 26597 catalog_manager.cc:4584] TS f902f2a1f1a24c98b0fccf205d84a5a5 (127.7.48.1:33555): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 938575a9c1ba4e38b9a39e727db39f93: Network error: Client connection negotiation failed: client connection to 127.7.48.1:33555: connect: Connection refused (error 111)
I20260928 06:49:13.081825 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 26941
I20260928 06:49:13.087015 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 26579
2026-09-28T06:49:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2431 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T06:49:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:13Z Disabled control of system clock
I20260928 06:49:13.116226 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:41371
--webserver_interface=127.7.48.62
--webserver_port=0
--builtin_ntp_servers=127.7.48.20:41537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:41371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:49:13.223670 27116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:13.224027 27116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:13.224093 27116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:13.227985 27116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:13.228082 27116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:13.228113 27116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:13.228152 27116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:13.228185 27116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:13.232821 27116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:41537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:41371
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:41371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:13.233973 27116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:13.235416 27116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:13.243672 27124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:13.243674 27121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:13.243732 27122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:13.244131 27116 server_base.cc:1034] running on GCE node
I20260928 06:49:13.244715 27116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:13.245824 27116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:13.247006 27116 hybrid_clock.cc:648] HybridClock initialized: now 1790578153246987 us; error 33 us; skew 500 ppm
I20260928 06:49:13.248963 27116 webserver.cc:460] Webserver started at http://127.7.48.62:37887/ using document root <none> and password file <none>
I20260928 06:49:13.249559 27116 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:13.249652 27116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:13.249883 27116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:13.251969 27116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "b9d0ab92702b48bc95ff4d2da3320de1"
format_stamp: "Formatted at 2026-09-28 06:49:13 on dist-test-slave-twl6"
server_key: "cd25c1ca4155c73333152d5b96c158cf"
server_key_iv: "d732686f0d93cee4a56f13c3bed0be81"
server_key_version: "encryptionkey@0"
I20260928 06:49:13.252532 27116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "b9d0ab92702b48bc95ff4d2da3320de1"
format_stamp: "Formatted at 2026-09-28 06:49:13 on dist-test-slave-twl6"
server_key: "cd25c1ca4155c73333152d5b96c158cf"
server_key_iv: "d732686f0d93cee4a56f13c3bed0be81"
server_key_version: "encryptionkey@0"
I20260928 06:49:13.256171 27116 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260928 06:49:13.258459 27130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:13.259687 27116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:49:13.259848 27116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "b9d0ab92702b48bc95ff4d2da3320de1"
format_stamp: "Formatted at 2026-09-28 06:49:13 on dist-test-slave-twl6"
server_key: "cd25c1ca4155c73333152d5b96c158cf"
server_key_iv: "d732686f0d93cee4a56f13c3bed0be81"
server_key_version: "encryptionkey@0"
I20260928 06:49:13.259984 27116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:13.279186 27116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:13.280046 27116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:13.280259 27116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:13.286995 27116 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:41371
I20260928 06:49:13.287014 27182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:41371 every 8 connection(s)
I20260928 06:49:13.288007 27116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:13.291178 27183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:13.291936 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 27116
I20260928 06:49:13.292071 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:13.292402 7360 external_mini_cluster.cc:1428] Setting key e70febe06b7fed19193f0771bceb72e5
I20260928 06:49:13.297513 27183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9d0ab92702b48bc95ff4d2da3320de1: Bootstrap starting.
I20260928 06:49:13.299695 27183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9d0ab92702b48bc95ff4d2da3320de1: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:13.300457 27183 log.cc:824] T 00000000000000000000000000000000 P b9d0ab92702b48bc95ff4d2da3320de1: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:13.301512 27186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:13.293910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56750 (local address 127.7.48.62:41371)
0928 06:49:13.294583 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:13.294608 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:13.294642 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:49:13.295294 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:13.295310 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:13.295714 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:13.296091 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:13.296501 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.297371 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.298305 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.298798 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.298966 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:13.299836 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:13.299842 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:13.299857 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:13.300264 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:13.300318 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:13.300339 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:13.300443 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:13.300579 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 06:49:13.300818 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":121,"threads_started":1}
I20260928 06:49:13.302541 27183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9d0ab92702b48bc95ff4d2da3320de1: No bootstrap required, opened a new log
I20260928 06:49:13.306025 27183 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9d0ab92702b48bc95ff4d2da3320de1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d0ab92702b48bc95ff4d2da3320de1" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 41371 } }
I20260928 06:49:13.306255 27183 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9d0ab92702b48bc95ff4d2da3320de1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:13.306327 27183 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9d0ab92702b48bc95ff4d2da3320de1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d0ab92702b48bc95ff4d2da3320de1, State: Initialized, Role: FOLLOWER
I20260928 06:49:13.306758 27183 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9d0ab92702b48bc95ff4d2da3320de1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d0ab92702b48bc95ff4d2da3320de1" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 41371 } }
I20260928 06:49:13.306900 27183 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9d0ab92702b48bc95ff4d2da3320de1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:49:13.306982 27183 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9d0ab92702b48bc95ff4d2da3320de1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:49:13.307075 27183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9d0ab92702b48bc95ff4d2da3320de1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:13.308081 27183 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9d0ab92702b48bc95ff4d2da3320de1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d0ab92702b48bc95ff4d2da3320de1" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 41371 } }
I20260928 06:49:13.308372 27183 leader_election.cc:302] T 00000000000000000000000000000000 P b9d0ab92702b48bc95ff4d2da3320de1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9d0ab92702b48bc95ff4d2da3320de1; no voters:
I20260928 06:49:13.308624 27183 leader_election.cc:288] T 00000000000000000000000000000000 P b9d0ab92702b48bc95ff4d2da3320de1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:49:13.308754 27188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9d0ab92702b48bc95ff4d2da3320de1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:13.309019 27188 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9d0ab92702b48bc95ff4d2da3320de1 [term 1 LEADER]: Becoming Leader. State: Replica: b9d0ab92702b48bc95ff4d2da3320de1, State: Running, Role: LEADER
I20260928 06:49:13.309306 27188 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9d0ab92702b48bc95ff4d2da3320de1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d0ab92702b48bc95ff4d2da3320de1" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 41371 } }
I20260928 06:49:13.309871 27183 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9d0ab92702b48bc95ff4d2da3320de1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:13.310968 27189 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d0ab92702b48bc95ff4d2da3320de1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9d0ab92702b48bc95ff4d2da3320de1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d0ab92702b48bc95ff4d2da3320de1" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 41371 } } }
I20260928 06:49:13.311007 27190 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d0ab92702b48bc95ff4d2da3320de1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9d0ab92702b48bc95ff4d2da3320de1. Latest consensus state: current_term: 1 leader_uuid: "b9d0ab92702b48bc95ff4d2da3320de1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d0ab92702b48bc95ff4d2da3320de1" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 41371 } } }
I20260928 06:49:13.311088 27189 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d0ab92702b48bc95ff4d2da3320de1 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:13.311115 27190 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d0ab92702b48bc95ff4d2da3320de1 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:13.311417 27193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:13.314399 27193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:49:13.319285 27193 catalog_manager.cc:1347] Generated new cluster ID: 8335f165ca3049b1a76df4058d51ede5
I20260928 06:49:13.319347 27193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:13.325582 27193 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:13.326437 27193 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:13.331664 27193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9d0ab92702b48bc95ff4d2da3320de1: Generated new TSK 0
I20260928 06:49:13.332348 27193 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:49:13.336534 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.1:0
--local_ip_for_outbound_sockets=127.7.48.1
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:41371
--builtin_ntp_servers=127.7.48.20:41537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:49:13.445348 27207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:13.445611 27207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:13.445648 27207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:13.449596 27207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:13.449652 27207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:13.449743 27207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.1
I20260928 06:49:13.454530 27207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:41537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.48.1
--webserver_port=0
--tserver_master_addrs=127.7.48.62:41371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.1
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:13.455611 27207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:13.457028 27207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:13.466387 27212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:13.466581 27207 server_base.cc:1034] running on GCE node
W20260928 06:49:13.466387 27213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:13.466445 27215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:13.467087 27207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:13.467718 27207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:13.468904 27207 hybrid_clock.cc:648] HybridClock initialized: now 1790578153468898 us; error 54 us; skew 500 ppm
I20260928 06:49:13.470687 27207 webserver.cc:460] Webserver started at http://127.7.48.1:40863/ using document root <none> and password file <none>
I20260928 06:49:13.471266 27207 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:13.471355 27207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:13.471585 27207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:13.473680 27207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-0/data/instance:
uuid: "9fd98004a22d46c59c3ca9459410a8bd"
format_stamp: "Formatted at 2026-09-28 06:49:13 on dist-test-slave-twl6"
server_key: "968e6ca5562e6ccae594d6d71b507541"
server_key_iv: "7b64ac41690ed33b90d050caefb194da"
server_key_version: "encryptionkey@0"
I20260928 06:49:13.474238 27207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance:
uuid: "9fd98004a22d46c59c3ca9459410a8bd"
format_stamp: "Formatted at 2026-09-28 06:49:13 on dist-test-slave-twl6"
server_key: "968e6ca5562e6ccae594d6d71b507541"
server_key_iv: "7b64ac41690ed33b90d050caefb194da"
server_key_version: "encryptionkey@0"
I20260928 06:49:13.477870 27207 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 06:49:13.480250 27221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:13.481423 27207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:49:13.481562 27207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-0/wal
uuid: "9fd98004a22d46c59c3ca9459410a8bd"
format_stamp: "Formatted at 2026-09-28 06:49:13 on dist-test-slave-twl6"
server_key: "968e6ca5562e6ccae594d6d71b507541"
server_key_iv: "7b64ac41690ed33b90d050caefb194da"
server_key_version: "encryptionkey@0"
I20260928 06:49:13.481671 27207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:13.502233 27207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:13.502889 27207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:13.503087 27207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:13.503664 27207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:49:13.504602 27207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:49:13.504666 27207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:13.504732 27207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:49:13.504769 27207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:13.513422 27207 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.1:32773
I20260928 06:49:13.513491 27334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.1:32773 every 8 connection(s)
I20260928 06:49:13.514350 27207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-0/data/info.pb
I20260928 06:49:13.522526 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 27207
I20260928 06:49:13.522657 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-0/wal/instance
I20260928 06:49:13.523015 7360 external_mini_cluster.cc:1428] Setting key bca4468f7c0446e0cfbefcfd317a5f6b
I20260928 06:49:13.523159 27186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:13.516114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:41515 (local address 127.7.48.62:41371)
0928 06:49:13.516247 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:13.516251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:13.517365 (+ 1114us) server_negotiation.cc:407] Connection header received
0928 06:49:13.518628 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:13.518631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:13.518675 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:13.518748 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:13.519568 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.520180 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.521250 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.521465 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.521537 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:13.522524 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:13.522527 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:13.522529 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:13.522541 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:13.522567 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:13.522623 (+ 56us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:13.522707 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:13.522921 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 06:49:13.522999 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 06:49:13.524138 27337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:13.516530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41371 (local address 127.7.48.1:41515)
0928 06:49:13.517221 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:13.517261 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:13.518439 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:13.518900 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:13.518906 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:13.518984 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:13.519443 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:13.519455 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.520307 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.520310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:13.521149 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:13.521158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.521587 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.521590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:13.521739 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:13.522267 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:49:13.522268 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:13.522305 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:49:13.522362 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:13.522749 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:13.522758 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:13.522764 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:49:13.522868 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 06:49:13.523109 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":139,"threads_started":1}
I20260928 06:49:13.525180 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.2:0
--local_ip_for_outbound_sockets=127.7.48.2
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:41371
--builtin_ntp_servers=127.7.48.20:41537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:49:13.525300 27335 heartbeater.cc:347] Connected to a master server at 127.7.48.62:41371
I20260928 06:49:13.525537 27335 heartbeater.cc:464] Registering TS with master...
I20260928 06:49:13.526012 27335 heartbeater.cc:511] Master 127.7.48.62:41371 requested a full tablet report, sending...
I20260928 06:49:13.527539 27147 ts_manager.cc:194] Registered new tserver with Master: 9fd98004a22d46c59c3ca9459410a8bd (127.7.48.1:32773)
I20260928 06:49:13.528800 27147 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.48.1:41515
W20260928 06:49:13.635990 27338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:13.636279 27338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:13.636358 27338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:13.641491 27338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:13.641561 27338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:13.641652 27338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.2
I20260928 06:49:13.646454 27338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:41537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.48.2
--webserver_port=0
--tserver_master_addrs=127.7.48.62:41371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.2
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:13.647578 27338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:13.649060 27338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:13.658473 27343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:13.658478 27344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:13.658516 27346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:13.658640 27338 server_base.cc:1034] running on GCE node
I20260928 06:49:13.659044 27338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:13.659663 27338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:13.660838 27338 hybrid_clock.cc:648] HybridClock initialized: now 1790578153660798 us; error 52 us; skew 500 ppm
I20260928 06:49:13.662843 27338 webserver.cc:460] Webserver started at http://127.7.48.2:45783/ using document root <none> and password file <none>
I20260928 06:49:13.663511 27338 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:13.663614 27338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:13.663846 27338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:13.665902 27338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-1/data/instance:
uuid: "a2208e92936f478084b8e2fb89f37c4f"
format_stamp: "Formatted at 2026-09-28 06:49:13 on dist-test-slave-twl6"
server_key: "2433952c0ff596db0557349de5a8cec5"
server_key_iv: "0dff88b1503988e8573ba73878ee706d"
server_key_version: "encryptionkey@0"
I20260928 06:49:13.666477 27338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance:
uuid: "a2208e92936f478084b8e2fb89f37c4f"
format_stamp: "Formatted at 2026-09-28 06:49:13 on dist-test-slave-twl6"
server_key: "2433952c0ff596db0557349de5a8cec5"
server_key_iv: "0dff88b1503988e8573ba73878ee706d"
server_key_version: "encryptionkey@0"
I20260928 06:49:13.670099 27338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 06:49:13.672602 27352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:13.673866 27338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 06:49:13.674001 27338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-1/wal
uuid: "a2208e92936f478084b8e2fb89f37c4f"
format_stamp: "Formatted at 2026-09-28 06:49:13 on dist-test-slave-twl6"
server_key: "2433952c0ff596db0557349de5a8cec5"
server_key_iv: "0dff88b1503988e8573ba73878ee706d"
server_key_version: "encryptionkey@0"
I20260928 06:49:13.674120 27338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:13.702335 27338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:13.703104 27338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:13.703308 27338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:13.703999 27338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:49:13.704947 27338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:49:13.705017 27338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:13.705088 27338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:49:13.705132 27338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:13.714489 27338 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.2:46443
I20260928 06:49:13.714519 27465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.2:46443 every 8 connection(s)
I20260928 06:49:13.715425 27338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-1/data/info.pb
I20260928 06:49:13.722193 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 27338
I20260928 06:49:13.722294 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-1/wal/instance
I20260928 06:49:13.722577 7360 external_mini_cluster.cc:1428] Setting key 0e19bf0625dfbcf12f7d1eb7cf82e4ef
I20260928 06:49:13.724922 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.48.3:0
--local_ip_for_outbound_sockets=127.7.48.3
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:41371
--builtin_ntp_servers=127.7.48.20:41537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:49:13.724984 27186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:13.717252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.2:60605 (local address 127.7.48.62:41371)
0928 06:49:13.717429 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:13.717434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:13.718516 (+ 1082us) server_negotiation.cc:407] Connection header received
0928 06:49:13.719846 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:13.719848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:13.719952 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:13.720038 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:13.720931 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.721447 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.722679 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.722982 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.723086 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:13.724476 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:13.724480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:13.724482 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:13.724495 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:13.724518 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:13.724528 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:13.724616 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:13.724759 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 06:49:13.724832 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 06:49:13.725899 27468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:13.717699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41371 (local address 127.7.48.2:60605)
0928 06:49:13.718355 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:13.718397 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:13.719659 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:13.720210 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:13.720217 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:13.720302 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:13.720804 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:13.720817 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.721559 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.721561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:13.722521 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:13.722530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.723166 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.723171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:13.723360 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:13.724179 (+ 819us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:49:13.724182 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:13.724242 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:49:13.724324 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:13.724616 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:13.724623 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:13.724628 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:49:13.724728 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 06:49:13.724956 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":116,"threads_started":1}
I20260928 06:49:13.727058 27466 heartbeater.cc:347] Connected to a master server at 127.7.48.62:41371
I20260928 06:49:13.727313 27466 heartbeater.cc:464] Registering TS with master...
I20260928 06:49:13.727856 27466 heartbeater.cc:511] Master 127.7.48.62:41371 requested a full tablet report, sending...
I20260928 06:49:13.728936 27147 ts_manager.cc:194] Registered new tserver with Master: a2208e92936f478084b8e2fb89f37c4f (127.7.48.2:46443)
I20260928 06:49:13.729516 27147 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.48.2:60605
W20260928 06:49:13.838222 27469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:13.838554 27469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:13.838644 27469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:13.842753 27469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:13.842834 27469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:13.842923 27469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.48.3
I20260928 06:49:13.847754 27469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:41537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.48.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.48.3
--webserver_port=0
--tserver_master_addrs=127.7.48.62:41371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.48.3
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:13.848886 27469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:13.850279 27469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:13.859572 27475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:13.859573 27477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:13.859573 27474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:13.859819 27469 server_base.cc:1034] running on GCE node
I20260928 06:49:13.860332 27469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:13.861073 27469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:13.862233 27469 hybrid_clock.cc:648] HybridClock initialized: now 1790578153862217 us; error 26 us; skew 500 ppm
I20260928 06:49:13.864578 27469 webserver.cc:460] Webserver started at http://127.7.48.3:43965/ using document root <none> and password file <none>
I20260928 06:49:13.865259 27469 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:13.865351 27469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:13.865568 27469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:13.867671 27469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-2/data/instance:
uuid: "df7ff1aaeacf40989fb00b3bef85775a"
format_stamp: "Formatted at 2026-09-28 06:49:13 on dist-test-slave-twl6"
server_key: "5bd14f6a34afeb2e7390a6c9ca4b1348"
server_key_iv: "ceeb497db47a00f3fb5330f3e361ade7"
server_key_version: "encryptionkey@0"
I20260928 06:49:13.868294 27469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance:
uuid: "df7ff1aaeacf40989fb00b3bef85775a"
format_stamp: "Formatted at 2026-09-28 06:49:13 on dist-test-slave-twl6"
server_key: "5bd14f6a34afeb2e7390a6c9ca4b1348"
server_key_iv: "ceeb497db47a00f3fb5330f3e361ade7"
server_key_version: "encryptionkey@0"
I20260928 06:49:13.872043 27469 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:49:13.874497 27483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:13.875761 27469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:49:13.875942 27469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-2/wal
uuid: "df7ff1aaeacf40989fb00b3bef85775a"
format_stamp: "Formatted at 2026-09-28 06:49:13 on dist-test-slave-twl6"
server_key: "5bd14f6a34afeb2e7390a6c9ca4b1348"
server_key_iv: "ceeb497db47a00f3fb5330f3e361ade7"
server_key_version: "encryptionkey@0"
I20260928 06:49:13.876072 27469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:13.886986 27469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:13.887657 27469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:13.887851 27469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:13.888475 27469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:49:13.889392 27469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:49:13.889468 27469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:13.889535 27469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:49:13.889590 27469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:13.898419 27469 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.3:41581
I20260928 06:49:13.898442 27596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.3:41581 every 8 connection(s)
I20260928 06:49:13.899360 27469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-2/data/info.pb
I20260928 06:49:13.901582 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 27469
I20260928 06:49:13.901660 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-2/wal/instance
I20260928 06:49:13.901923 7360 external_mini_cluster.cc:1428] Setting key 71fb65401e85c10459ba8ce3e0613962
I20260928 06:49:13.908360 27186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:13.901197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.3:39777 (local address 127.7.48.62:41371)
0928 06:49:13.901360 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:13.901365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:13.902436 (+ 1071us) server_negotiation.cc:407] Connection header received
0928 06:49:13.904150 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:13.904153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:13.904207 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:13.904271 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:13.905095 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.905598 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.906627 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.906824 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.906894 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:13.907866 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:13.907869 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:13.907871 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:13.907925 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:13.907956 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:13.907964 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:13.908025 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:13.908168 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 06:49:13.908216 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:49:13.909142 27599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:13.901568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:41371 (local address 127.7.48.3:39777)
0928 06:49:13.902284 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:13.902335 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:13.903951 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:13.904434 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:13.904440 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:13.904515 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:13.904979 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:13.904991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.905704 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.905706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:13.906519 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:13.906528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.906944 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.906947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:13.907078 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:13.907637 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:49:13.907638 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:13.907677 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:49:13.907735 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:13.908044 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:13.908051 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:13.908055 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:49:13.908151 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 06:49:13.908350 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":139,"threads_started":1}
I20260928 06:49:13.910230 27597 heartbeater.cc:347] Connected to a master server at 127.7.48.62:41371
I20260928 06:49:13.910506 27597 heartbeater.cc:464] Registering TS with master...
I20260928 06:49:13.911065 27597 heartbeater.cc:511] Master 127.7.48.62:41371 requested a full tablet report, sending...
I20260928 06:49:13.912112 27147 ts_manager.cc:194] Registered new tserver with Master: df7ff1aaeacf40989fb00b3bef85775a (127.7.48.3:41581)
I20260928 06:49:13.912680 27147 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.48.3:39777
I20260928 06:49:13.915719 7360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:49:13.921377 27186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:13.917609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56758 (local address 127.7.48.62:41371)
0928 06:49:13.917731 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:13.917736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:13.917915 (+ 179us) server_negotiation.cc:407] Connection header received
0928 06:49:13.918071 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:13.918074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:13.918120 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:13.918200 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:13.918473 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.918955 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.919779 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.920025 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.920135 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:13.920948 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:13.920951 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:13.920954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:13.920978 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:13.920996 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:13.921004 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:13.921081 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:13.921196 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 06:49:13.921245 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:49:13.924158 27147 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:49:13.927007 27147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:49:13.940582 27608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:13.934968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:41581 (local address 127.0.0.1:40942)
0928 06:49:13.935356 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:13.935427 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:13.935556 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:13.936348 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:13.936354 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:13.936378 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:13.936518 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:13.936528 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.937836 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.937841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:13.938918 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:13.938927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.940027 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.940031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:13.940135 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:13.940176 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 06:49:13.940299 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 06:49:13.940413 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":97,"threads_started":1}
I20260928 06:49:13.940605 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:13.935149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40942 (local address 127.7.48.3:41581)
0928 06:49:13.935507 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:13.935520 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:13.935544 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:49:13.935737 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:13.935744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:13.935989 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:13.936151 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:13.936662 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.937729 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.939054 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.939837 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.940004 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:13.940211 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:13.940327 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 06:49:13.940434 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":116,"threads_started":1}
I20260928 06:49:13.941489 27610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:13.936102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:46443 (local address 127.0.0.1:36612)
0928 06:49:13.936508 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:13.936521 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:13.936612 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:13.937283 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:13.937286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:13.937297 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:13.937445 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:13.937452 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.938780 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.938784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:13.940023 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:13.940034 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.941110 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.941114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:13.941195 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:13.941217 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 06:49:13.941283 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:49:13.941346 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":43,"thread_start_us":103,"threads_started":1}
I20260928 06:49:13.941541 27613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:13.936525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.7.48.2:46443)
0928 06:49:13.936872 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:13.936884 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:13.936903 (+ 19us) server_negotiation.cc:407] Connection header received
0928 06:49:13.936963 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:13.936967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:13.937108 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:13.937364 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:13.937545 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.938661 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.940160 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.940943 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.941093 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:13.941268 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:13.941315 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 06:49:13.941397 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":114,"threads_started":1}
I20260928 06:49:13.942684 27612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:13.935735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:32773 (local address 127.0.0.1:42874)
0928 06:49:13.936410 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:13.936422 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:13.936497 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:13.937216 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:13.937222 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:13.937244 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:13.937366 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:13.937372 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.938780 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.938784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:13.941256 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:13.941266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.942310 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.942314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:13.942401 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:13.942422 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 06:49:13.942468 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:49:13.942534 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":6,"thread_start_us":98,"threads_started":1}
I20260928 06:49:13.943349 27611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:13.935997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42874 (local address 127.7.48.1:32773)
0928 06:49:13.936739 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:13.936752 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:13.936772 (+ 20us) server_negotiation.cc:407] Connection header received
0928 06:49:13.936858 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:13.936862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:13.937033 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:13.937212 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:13.937496 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.938638 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.941391 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:13.942170 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:13.942986 (+ 816us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:13.943124 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:13.943163 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 06:49:13.943230 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":137,"threads_started":1}
I20260928 06:49:13.944041 27531 tablet_service.cc:1467] Processing CreateTablet for tablet 0531c84f81c7496aa0c7c04dcb984762 (DEFAULT_TABLE table=test-table [id=79520fa94efb4d49ac72f56581bcbd54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:49:13.944561 27400 tablet_service.cc:1467] Processing CreateTablet for tablet 0531c84f81c7496aa0c7c04dcb984762 (DEFAULT_TABLE table=test-table [id=79520fa94efb4d49ac72f56581bcbd54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:49:13.945233 27531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0531c84f81c7496aa0c7c04dcb984762. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:13.945603 27400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0531c84f81c7496aa0c7c04dcb984762. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:13.945663 27269 tablet_service.cc:1467] Processing CreateTablet for tablet 0531c84f81c7496aa0c7c04dcb984762 (DEFAULT_TABLE table=test-table [id=79520fa94efb4d49ac72f56581bcbd54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:49:13.946578 27269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0531c84f81c7496aa0c7c04dcb984762. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:13.952708 27614 tablet_bootstrap.cc:492] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd: Bootstrap starting.
I20260928 06:49:13.953572 27615 tablet_bootstrap.cc:492] T 0531c84f81c7496aa0c7c04dcb984762 P a2208e92936f478084b8e2fb89f37c4f: Bootstrap starting.
I20260928 06:49:13.955130 27614 tablet_bootstrap.cc:654] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:13.955333 27615 tablet_bootstrap.cc:654] T 0531c84f81c7496aa0c7c04dcb984762 P a2208e92936f478084b8e2fb89f37c4f: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:13.956029 27614 log.cc:824] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:13.956084 27615 log.cc:824] T 0531c84f81c7496aa0c7c04dcb984762 P a2208e92936f478084b8e2fb89f37c4f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:13.956283 27616 tablet_bootstrap.cc:492] T 0531c84f81c7496aa0c7c04dcb984762 P df7ff1aaeacf40989fb00b3bef85775a: Bootstrap starting.
I20260928 06:49:13.958508 27616 tablet_bootstrap.cc:654] T 0531c84f81c7496aa0c7c04dcb984762 P df7ff1aaeacf40989fb00b3bef85775a: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:13.958550 27614 tablet_bootstrap.cc:492] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd: No bootstrap required, opened a new log
I20260928 06:49:13.958783 27614 ts_tablet_manager.cc:1397] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 06:49:13.958834 27615 tablet_bootstrap.cc:492] T 0531c84f81c7496aa0c7c04dcb984762 P a2208e92936f478084b8e2fb89f37c4f: No bootstrap required, opened a new log
I20260928 06:49:13.959004 27615 ts_tablet_manager.cc:1397] T 0531c84f81c7496aa0c7c04dcb984762 P a2208e92936f478084b8e2fb89f37c4f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260928 06:49:13.959448 27616 log.cc:824] T 0531c84f81c7496aa0c7c04dcb984762 P df7ff1aaeacf40989fb00b3bef85775a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:13.961561 27616 tablet_bootstrap.cc:492] T 0531c84f81c7496aa0c7c04dcb984762 P df7ff1aaeacf40989fb00b3bef85775a: No bootstrap required, opened a new log
I20260928 06:49:13.961740 27616 ts_tablet_manager.cc:1397] T 0531c84f81c7496aa0c7c04dcb984762 P df7ff1aaeacf40989fb00b3bef85775a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 06:49:13.962643 27614 raft_consensus.cc:348] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7ff1aaeacf40989fb00b3bef85775a" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41581 } } peers { permanent_uuid: "9fd98004a22d46c59c3ca9459410a8bd" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 32773 } } peers { permanent_uuid: "a2208e92936f478084b8e2fb89f37c4f" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 46443 } }
I20260928 06:49:13.962956 27614 raft_consensus.cc:374] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:13.963057 27614 raft_consensus.cc:729] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fd98004a22d46c59c3ca9459410a8bd, State: Initialized, Role: FOLLOWER
I20260928 06:49:13.963393 27615 raft_consensus.cc:348] T 0531c84f81c7496aa0c7c04dcb984762 P a2208e92936f478084b8e2fb89f37c4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7ff1aaeacf40989fb00b3bef85775a" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41581 } } peers { permanent_uuid: "9fd98004a22d46c59c3ca9459410a8bd" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 32773 } } peers { permanent_uuid: "a2208e92936f478084b8e2fb89f37c4f" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 46443 } }
I20260928 06:49:13.963675 27615 raft_consensus.cc:374] T 0531c84f81c7496aa0c7c04dcb984762 P a2208e92936f478084b8e2fb89f37c4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:13.963724 27615 raft_consensus.cc:729] T 0531c84f81c7496aa0c7c04dcb984762 P a2208e92936f478084b8e2fb89f37c4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2208e92936f478084b8e2fb89f37c4f, State: Initialized, Role: FOLLOWER
I20260928 06:49:13.963613 27614 consensus_queue.cc:260] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7ff1aaeacf40989fb00b3bef85775a" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41581 } } peers { permanent_uuid: "9fd98004a22d46c59c3ca9459410a8bd" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 32773 } } peers { permanent_uuid: "a2208e92936f478084b8e2fb89f37c4f" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 46443 } }
I20260928 06:49:13.964179 27615 consensus_queue.cc:260] T 0531c84f81c7496aa0c7c04dcb984762 P a2208e92936f478084b8e2fb89f37c4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7ff1aaeacf40989fb00b3bef85775a" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41581 } } peers { permanent_uuid: "9fd98004a22d46c59c3ca9459410a8bd" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 32773 } } peers { permanent_uuid: "a2208e92936f478084b8e2fb89f37c4f" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 46443 } }
I20260928 06:49:13.964803 27614 ts_tablet_manager.cc:1428] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 06:49:13.965132 27615 ts_tablet_manager.cc:1428] T 0531c84f81c7496aa0c7c04dcb984762 P a2208e92936f478084b8e2fb89f37c4f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 06:49:13.965198 27335 heartbeater.cc:503] Master 127.7.48.62:41371 was elected leader, sending a full tablet report...
I20260928 06:49:13.965502 27466 heartbeater.cc:503] Master 127.7.48.62:41371 was elected leader, sending a full tablet report...
I20260928 06:49:13.965969 27616 raft_consensus.cc:348] T 0531c84f81c7496aa0c7c04dcb984762 P df7ff1aaeacf40989fb00b3bef85775a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7ff1aaeacf40989fb00b3bef85775a" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41581 } } peers { permanent_uuid: "9fd98004a22d46c59c3ca9459410a8bd" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 32773 } } peers { permanent_uuid: "a2208e92936f478084b8e2fb89f37c4f" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 46443 } }
W20260928 06:49:13.966359 27467 tablet.cc:2369] T 0531c84f81c7496aa0c7c04dcb984762 P a2208e92936f478084b8e2fb89f37c4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:49:13.966562 27616 raft_consensus.cc:374] T 0531c84f81c7496aa0c7c04dcb984762 P df7ff1aaeacf40989fb00b3bef85775a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:13.966621 27616 raft_consensus.cc:729] T 0531c84f81c7496aa0c7c04dcb984762 P df7ff1aaeacf40989fb00b3bef85775a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df7ff1aaeacf40989fb00b3bef85775a, State: Initialized, Role: FOLLOWER
I20260928 06:49:13.967023 27616 consensus_queue.cc:260] T 0531c84f81c7496aa0c7c04dcb984762 P df7ff1aaeacf40989fb00b3bef85775a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7ff1aaeacf40989fb00b3bef85775a" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41581 } } peers { permanent_uuid: "9fd98004a22d46c59c3ca9459410a8bd" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 32773 } } peers { permanent_uuid: "a2208e92936f478084b8e2fb89f37c4f" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 46443 } }
I20260928 06:49:13.967679 27597 heartbeater.cc:503] Master 127.7.48.62:41371 was elected leader, sending a full tablet report...
I20260928 06:49:13.968045 27616 ts_tablet_manager.cc:1428] T 0531c84f81c7496aa0c7c04dcb984762 P df7ff1aaeacf40989fb00b3bef85775a: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
W20260928 06:49:14.015522 27336 tablet.cc:2369] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:49:14.150451 27598 tablet.cc:2369] T 0531c84f81c7496aa0c7c04dcb984762 P df7ff1aaeacf40989fb00b3bef85775a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:49:14.161520 27620 raft_consensus.cc:482] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:49:14.161819 27620 raft_consensus.cc:504] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7ff1aaeacf40989fb00b3bef85775a" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41581 } } peers { permanent_uuid: "9fd98004a22d46c59c3ca9459410a8bd" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 32773 } } peers { permanent_uuid: "a2208e92936f478084b8e2fb89f37c4f" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 46443 } }
I20260928 06:49:14.163015 27620 leader_election.cc:288] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df7ff1aaeacf40989fb00b3bef85775a (127.7.48.3:41581), a2208e92936f478084b8e2fb89f37c4f (127.7.48.2:46443)
I20260928 06:49:14.167352 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:14.163442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:56867 (local address 127.7.48.3:41581)
0928 06:49:14.163588 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:14.163593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:14.163809 (+ 216us) server_negotiation.cc:407] Connection header received
0928 06:49:14.164125 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:14.164129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:14.164190 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:14.164241 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:14.164774 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:14.165256 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:14.166420 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:14.166809 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:14.166926 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:14.166989 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:14.167160 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 06:49:14.167230 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:49:14.167356 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:14.163327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.3:41581 (local address 127.7.48.1:56867)
0928 06:49:14.163684 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:14.163698 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:14.163959 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:14.164388 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:14.164390 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:14.164454 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:14.164659 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:14.164665 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:14.165352 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:14.165355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:14.166301 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:14.166311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:14.166931 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:14.166937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:14.167037 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:14.167056 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 06:49:14.167121 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:49:14.167172 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":33,"spinlock_wait_cycles":768,"thread_start_us":117,"threads_started":1}
I20260928 06:49:14.168314 27624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:14.163797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:46443 (local address 127.7.48.1:42303)
0928 06:49:14.164113 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:14.164126 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:14.164205 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:14.164454 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:14.164456 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:14.164467 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:49:14.164653 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:14.164660 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:14.165941 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:14.165944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:14.166811 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:14.166819 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:14.167870 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:14.167874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:14.167992 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:14.168015 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 06:49:14.168102 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:49:14.168173 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":32,"thread_start_us":90,"threads_started":1}
I20260928 06:49:14.168231 27551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0531c84f81c7496aa0c7c04dcb984762" candidate_uuid: "9fd98004a22d46c59c3ca9459410a8bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df7ff1aaeacf40989fb00b3bef85775a" is_pre_election: true
I20260928 06:49:14.168337 27613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:14.163998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.48.1:42303 (local address 127.7.48.2:46443)
0928 06:49:14.164129 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:14.164138 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:14.164175 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:49:14.164307 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:14.164309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:14.164352 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:14.164417 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:49:14.165381 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:14.165831 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:14.166958 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:14.167573 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:14.167673 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:14.167729 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:14.168149 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 06:49:14.168222 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:49:14.168578 27551 raft_consensus.cc:2454] T 0531c84f81c7496aa0c7c04dcb984762 P df7ff1aaeacf40989fb00b3bef85775a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fd98004a22d46c59c3ca9459410a8bd in term 0.
I20260928 06:49:14.169054 27420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0531c84f81c7496aa0c7c04dcb984762" candidate_uuid: "9fd98004a22d46c59c3ca9459410a8bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2208e92936f478084b8e2fb89f37c4f" is_pre_election: true
I20260928 06:49:14.169076 27224 leader_election.cc:302] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fd98004a22d46c59c3ca9459410a8bd, df7ff1aaeacf40989fb00b3bef85775a; no voters:
I20260928 06:49:14.169304 27420 raft_consensus.cc:2454] T 0531c84f81c7496aa0c7c04dcb984762 P a2208e92936f478084b8e2fb89f37c4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fd98004a22d46c59c3ca9459410a8bd in term 0.
I20260928 06:49:14.169353 27620 raft_consensus.cc:2781] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:49:14.169428 27620 raft_consensus.cc:482] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:49:14.169478 27620 raft_consensus.cc:3037] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:14.170527 27620 raft_consensus.cc:504] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7ff1aaeacf40989fb00b3bef85775a" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41581 } } peers { permanent_uuid: "9fd98004a22d46c59c3ca9459410a8bd" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 32773 } } peers { permanent_uuid: "a2208e92936f478084b8e2fb89f37c4f" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 46443 } }
I20260928 06:49:14.170892 27620 leader_election.cc:288] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd [CANDIDATE]: Term 1 election: Requested vote from peers df7ff1aaeacf40989fb00b3bef85775a (127.7.48.3:41581), a2208e92936f478084b8e2fb89f37c4f (127.7.48.2:46443)
I20260928 06:49:14.171211 27551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0531c84f81c7496aa0c7c04dcb984762" candidate_uuid: "9fd98004a22d46c59c3ca9459410a8bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df7ff1aaeacf40989fb00b3bef85775a"
I20260928 06:49:14.171232 27420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0531c84f81c7496aa0c7c04dcb984762" candidate_uuid: "9fd98004a22d46c59c3ca9459410a8bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2208e92936f478084b8e2fb89f37c4f"
I20260928 06:49:14.171319 27551 raft_consensus.cc:3037] T 0531c84f81c7496aa0c7c04dcb984762 P df7ff1aaeacf40989fb00b3bef85775a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:14.171352 27420 raft_consensus.cc:3037] T 0531c84f81c7496aa0c7c04dcb984762 P a2208e92936f478084b8e2fb89f37c4f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:14.172366 27551 raft_consensus.cc:2454] T 0531c84f81c7496aa0c7c04dcb984762 P df7ff1aaeacf40989fb00b3bef85775a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fd98004a22d46c59c3ca9459410a8bd in term 1.
I20260928 06:49:14.172680 27224 leader_election.cc:302] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fd98004a22d46c59c3ca9459410a8bd, df7ff1aaeacf40989fb00b3bef85775a; no voters:
I20260928 06:49:14.172693 27420 raft_consensus.cc:2454] T 0531c84f81c7496aa0c7c04dcb984762 P a2208e92936f478084b8e2fb89f37c4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fd98004a22d46c59c3ca9459410a8bd in term 1.
I20260928 06:49:14.172837 27620 raft_consensus.cc:2781] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:14.173071 27620 raft_consensus.cc:686] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd [term 1 LEADER]: Becoming Leader. State: Replica: 9fd98004a22d46c59c3ca9459410a8bd, State: Running, Role: LEADER
I20260928 06:49:14.173408 27620 consensus_queue.cc:237] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7ff1aaeacf40989fb00b3bef85775a" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41581 } } peers { permanent_uuid: "9fd98004a22d46c59c3ca9459410a8bd" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 32773 } } peers { permanent_uuid: "a2208e92936f478084b8e2fb89f37c4f" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 46443 } }
I20260928 06:49:14.176523 27146 catalog_manager.cc:5495] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fd98004a22d46c59c3ca9459410a8bd (127.7.48.1). New cstate: current_term: 1 leader_uuid: "9fd98004a22d46c59c3ca9459410a8bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7ff1aaeacf40989fb00b3bef85775a" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fd98004a22d46c59c3ca9459410a8bd" member_type: VOTER last_known_addr { host: "127.7.48.1" port: 32773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2208e92936f478084b8e2fb89f37c4f" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 46443 } health_report { overall_health: UNKNOWN } } }
I20260928 06:49:14.232761 27611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:14.229706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42888 (local address 127.7.48.1:32773)
0928 06:49:14.229837 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:14.229841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:14.229853 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:49:14.229929 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:14.229932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:14.229989 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:14.230088 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:49:14.230281 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:14.230756 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:14.231479 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:14.231668 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:14.231737 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:14.232022 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:49:14.232442 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:49:14.232531 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:14.232582 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 06:49:14.232631 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:49:14.238794 27420 raft_consensus.cc:1261] T 0531c84f81c7496aa0c7c04dcb984762 P a2208e92936f478084b8e2fb89f37c4f [term 1 FOLLOWER]: Refusing update from remote peer 9fd98004a22d46c59c3ca9459410a8bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:49:14.238794 27551 raft_consensus.cc:1261] T 0531c84f81c7496aa0c7c04dcb984762 P df7ff1aaeacf40989fb00b3bef85775a [term 1 FOLLOWER]: Refusing update from remote peer 9fd98004a22d46c59c3ca9459410a8bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:49:14.239516 27620 consensus_queue.cc:1035] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2208e92936f478084b8e2fb89f37c4f" member_type: VOTER last_known_addr { host: "127.7.48.2" port: 46443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:49:14.239706 27625 consensus_queue.cc:1035] T 0531c84f81c7496aa0c7c04dcb984762 P 9fd98004a22d46c59c3ca9459410a8bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "df7ff1aaeacf40989fb00b3bef85775a" member_type: VOTER last_known_addr { host: "127.7.48.3" port: 41581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:49:14.257333 27146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:49:14.257488 27146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:49:14.260061 27146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b9d0ab92702b48bc95ff4d2da3320de1: Sending DeleteTablet for 3 replicas of tablet 0531c84f81c7496aa0c7c04dcb984762
I20260928 06:49:14.260936 27531 tablet_service.cc:1514] Processing DeleteTablet for tablet 0531c84f81c7496aa0c7c04dcb984762 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:49:14 UTC) from {username='slave'} at 127.0.0.1:40942
I20260928 06:49:14.261019 27269 tablet_service.cc:1514] Processing DeleteTablet for tablet 0531c84f81c7496aa0c7c04dcb984762 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:49:14 UTC) from {username='slave'} at 127.0.0.1:42874
I20260928 06:49:14.261066 27400 tablet_service.cc:1514] Processing DeleteTablet for tablet 0531c84f81c7496aa0c7c04dcb984762 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:49:14 UTC) from {username='slave'} at 127.0.0.1:36612
I20260928 06:49:14.261691 27639 tablet_replica.cc:330] T 0531c84f81c7496aa0c7c04dcb984762 P a2208e92936f478084b8e2fb89f37c4f: stopping tablet replica
I20260928 06:49:14.261899 27638 tablet_replica.cc:330] T 0531c84f81c7496aa0c7c04dcb984762 P df7ff1aaeacf40989fb00b3bef85775a: stopping tablet replica
I20260928 06:49:14.262071 27639 raft_consensus.cc:2229] T 0531c84f81c7496aa0c7c04dcb984762 P a2208e92936f478084b8e2fb89f37c4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:49:14.262346 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 27207
I20260928 06:49:14.263239 27639 raft_consensus.cc:2258] T 0531c84f81c7496aa0c7c04dcb984762 P a2208e92936f478084b8e2fb89f37c4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:49:14.264154 27638 raft_consensus.cc:2229] T 0531c84f81c7496aa0c7c04dcb984762 P df7ff1aaeacf40989fb00b3bef85775a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:49:14.264529 27638 raft_consensus.cc:2258] T 0531c84f81c7496aa0c7c04dcb984762 P df7ff1aaeacf40989fb00b3bef85775a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:49:14.265904 27638 ts_tablet_manager.cc:1905] T 0531c84f81c7496aa0c7c04dcb984762 P df7ff1aaeacf40989fb00b3bef85775a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:49:14.268162 27638 ts_tablet_manager.cc:1918] T 0531c84f81c7496aa0c7c04dcb984762 P df7ff1aaeacf40989fb00b3bef85775a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:49:14.268249 27638 log.cc:1196] T 0531c84f81c7496aa0c7c04dcb984762 P df7ff1aaeacf40989fb00b3bef85775a: Deleting WAL directory at /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577905526996-7360-0/minicluster-data/ts-2/wal/wals/0531c84f81c7496aa0c7c04dcb984762
I20260928 06:49:14.268615 27638 ts_tablet_manager.cc:1939] T 0531c84f81c7496aa0c7c04dcb984762 P df7ff1aaeacf40989fb00b3bef85775a: Deleting consensus metadata
I20260928 06:49:14.269985 27133 catalog_manager.cc:4841] TS df7ff1aaeacf40989fb00b3bef85775a (127.7.48.3:41581): tablet 0531c84f81c7496aa0c7c04dcb984762 (table test-table [id=79520fa94efb4d49ac72f56581bcbd54]) successfully deleted
I20260928 06:49:14.270447 27639 ts_tablet_manager.cc:1905] T 0531c84f81c7496aa0c7c04dcb984762 P a2208e92936f478084b8e2fb89f37c4f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:49:14.272152 27133 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.48.1:32773 (error 108)
I20260928 06:49:14.272413 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 27338
I20260928 06:49:14.276819 27612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:14.276432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.1:32773 (local address 127.0.0.1:42890)
0928 06:49:14.276605 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:14.276703 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.1:32773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 06:49:14.277017 27133 catalog_manager.cc:4584] TS 9fd98004a22d46c59c3ca9459410a8bd (127.7.48.1:32773): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0531c84f81c7496aa0c7c04dcb984762: Network error: Client connection negotiation failed: client connection to 127.7.48.1:32773: connect: Connection refused (error 111)
I20260928 06:49:14.278650 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 27469
I20260928 06:49:14.279541 27612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:14.278713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.2:46443 (local address 127.0.0.1:36618)
0928 06:49:14.279386 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:14.279447 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.2:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580}
I20260928 06:49:14.284713 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 27116
2026-09-28T06:49:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1198 ms)
[----------] 6 tests from EncryptionPolicyTest (10164 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:49:14 dist-test-slave-twl6 krb5kdc[27644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:49:14 dist-test-slave-twl6 krb5kdc[27644](info): set up 2 sockets
Sep 28 06:49:14 dist-test-slave-twl6 krb5kdc[27644](info): commencing operation
krb5kdc: starting...
W20260928 06:49:15.300384 7360 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:49:15 dist-test-slave-twl6 krb5kdc[27644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:49:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:49:15.423967 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34515
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:15.535774 27657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:15.536170 27657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:15.536263 27657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:15.540287 27657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:15.540397 27657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:15.540436 27657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:15.540473 27657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:15.540500 27657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:15.545855 27657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34515
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:15.547052 27657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:15.548197 27657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:15.556958 27663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:15.556960 27662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:15.557051 27657 server_base.cc:1034] running on GCE node
W20260928 06:49:15.556960 27665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:15.557889 27657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:15.559069 27657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:15.560277 27657 hybrid_clock.cc:648] HybridClock initialized: now 1790578155560264 us; error 35 us; skew 500 ppm
Sep 28 06:49:15 dist-test-slave-twl6 krb5kdc[27644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:15.563025 27657 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:49:15.564088 27657 webserver.cc:460] Webserver started at http://127.0.0.1:39951/ using document root <none> and password file <none>
I20260928 06:49:15.564793 27657 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:15.564885 27657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:15.565104 27657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:15.567202 27657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "6b1a9cdbf8a541d08a9e45c666c028d0"
format_stamp: "Formatted at 2026-09-28 06:49:15 on dist-test-slave-twl6"
server_key: "2fd496fd65d4cce2c17a940ece61a0c2"
server_key_iv: "e3cc2750a43bdb34ca73a880608670de"
server_key_version: "encryptionkey@0"
I20260928 06:49:15.567780 27657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "6b1a9cdbf8a541d08a9e45c666c028d0"
format_stamp: "Formatted at 2026-09-28 06:49:15 on dist-test-slave-twl6"
server_key: "2fd496fd65d4cce2c17a940ece61a0c2"
server_key_iv: "e3cc2750a43bdb34ca73a880608670de"
server_key_version: "encryptionkey@0"
I20260928 06:49:15.571775 27657 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 06:49:15.574376 27672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:15.575611 27657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:49:15.575757 27657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "6b1a9cdbf8a541d08a9e45c666c028d0"
format_stamp: "Formatted at 2026-09-28 06:49:15 on dist-test-slave-twl6"
server_key: "2fd496fd65d4cce2c17a940ece61a0c2"
server_key_iv: "e3cc2750a43bdb34ca73a880608670de"
server_key_version: "encryptionkey@0"
I20260928 06:49:15.575881 27657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:15.596164 27657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:15.599119 27657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:15.599334 27657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:15.606807 27657 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34515
I20260928 06:49:15.606827 27724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34515 every 8 connection(s)
I20260928 06:49:15.607811 27657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:15.610652 27725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:15.610903 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 27657
I20260928 06:49:15.611052 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:15.611363 7360 external_mini_cluster.cc:1428] Setting key 05febcd74ffee6c8eb50be24e44b8ae8
I20260928 06:49:15.617431 27725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b1a9cdbf8a541d08a9e45c666c028d0: Bootstrap starting.
I20260928 06:49:15.619952 27725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b1a9cdbf8a541d08a9e45c666c028d0: Neither blocks nor log segments found. Creating new log.
Sep 28 06:49:15 dist-test-slave-twl6 krb5kdc[27644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:49:15.620900 27725 log.cc:824] T 00000000000000000000000000000000 P 6b1a9cdbf8a541d08a9e45c666c028d0: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:15.623176 27725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b1a9cdbf8a541d08a9e45c666c028d0: No bootstrap required, opened a new log
I20260928 06:49:15.626103 27728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:15.612904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45877 (local address 127.0.0.1:34515)
0928 06:49:15.613360 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:15.613377 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:15.613413 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:49:15.614111 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:15.614130 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:15.614484 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:15.614844 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:15.615650 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:15.616724 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:15.617635 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:15.618156 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:15.618329 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:15.621210 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:15.621216 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:15.621229 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:15.621255 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:15.623050 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:15.623667 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:15.623673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:15.623678 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:15.623787 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:15.624190 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:15.624192 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:15.624193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:15.624604 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:15.624775 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:15.625093 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 06:49:15.625323 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":131,"threads_started":1}
I20260928 06:49:15.627547 27725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b1a9cdbf8a541d08a9e45c666c028d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1a9cdbf8a541d08a9e45c666c028d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34515 } }
I20260928 06:49:15.627810 27725 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b1a9cdbf8a541d08a9e45c666c028d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:15.627910 27725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b1a9cdbf8a541d08a9e45c666c028d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1a9cdbf8a541d08a9e45c666c028d0, State: Initialized, Role: FOLLOWER
I20260928 06:49:15.628412 27725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b1a9cdbf8a541d08a9e45c666c028d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1a9cdbf8a541d08a9e45c666c028d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34515 } }
I20260928 06:49:15.628582 27725 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b1a9cdbf8a541d08a9e45c666c028d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:49:15.628670 27725 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b1a9cdbf8a541d08a9e45c666c028d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:49:15.628791 27725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b1a9cdbf8a541d08a9e45c666c028d0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:15.629310 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:49:15.629809 27725 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b1a9cdbf8a541d08a9e45c666c028d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1a9cdbf8a541d08a9e45c666c028d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34515 } }
I20260928 06:49:15.630144 27725 leader_election.cc:302] T 00000000000000000000000000000000 P 6b1a9cdbf8a541d08a9e45c666c028d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b1a9cdbf8a541d08a9e45c666c028d0; no voters:
I20260928 06:49:15.630484 27725 leader_election.cc:288] T 00000000000000000000000000000000 P 6b1a9cdbf8a541d08a9e45c666c028d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:49:15.630622 27731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b1a9cdbf8a541d08a9e45c666c028d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:15.630915 27731 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b1a9cdbf8a541d08a9e45c666c028d0 [term 1 LEADER]: Becoming Leader. State: Replica: 6b1a9cdbf8a541d08a9e45c666c028d0, State: Running, Role: LEADER
I20260928 06:49:15.631282 27731 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b1a9cdbf8a541d08a9e45c666c028d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1a9cdbf8a541d08a9e45c666c028d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34515 } }
I20260928 06:49:15.631945 27725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b1a9cdbf8a541d08a9e45c666c028d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:15.633170 27736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b1a9cdbf8a541d08a9e45c666c028d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b1a9cdbf8a541d08a9e45c666c028d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1a9cdbf8a541d08a9e45c666c028d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34515 } } }
I20260928 06:49:15.633442 27736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b1a9cdbf8a541d08a9e45c666c028d0 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:15.633737 27736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b1a9cdbf8a541d08a9e45c666c028d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b1a9cdbf8a541d08a9e45c666c028d0. Latest consensus state: current_term: 1 leader_uuid: "6b1a9cdbf8a541d08a9e45c666c028d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1a9cdbf8a541d08a9e45c666c028d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34515 } } }
I20260928 06:49:15.633879 27736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b1a9cdbf8a541d08a9e45c666c028d0 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:15.634862 27746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:15.637661 27746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:49:15.640990 27728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:15.631462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45355 (local address 127.0.0.1:34515)
0928 06:49:15.631566 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:15.631570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:15.631814 (+ 244us) server_negotiation.cc:407] Connection header received
0928 06:49:15.631969 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:15.631972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:15.632023 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:15.632111 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:15.632987 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:15.633854 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:15.635105 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:15.635362 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:15.635471 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:15.637186 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:15.637192 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:15.637195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:15.637220 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:15.638993 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:15.639574 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:15.639578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:15.639579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:15.639668 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:15.640060 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:15.640063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:15.640065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:15.640278 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:15.640424 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:15.640749 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 06:49:15.640846 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":38}
I20260928 06:49:15.643908 27746 catalog_manager.cc:1347] Generated new cluster ID: 4be07d51c58949d4b52ca9b624484b03
I20260928 06:49:15.644016 27746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:15.669607 27746 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:15.670575 27746 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:15.677449 27746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b1a9cdbf8a541d08a9e45c666c028d0: Generated new TSK 0
I20260928 06:49:15.678375 27746 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:49:15.688423 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 27657
W20260928 06:49:15.693506 27642 connection.cc:535] client connection to 127.0.0.1:34515 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34515): Connection reset by peer (error 104)
2026-09-28T06:49:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:49:15 dist-test-slave-twl6 krb5kdc[27756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:49:15 dist-test-slave-twl6 krb5kdc[27756](info): set up 2 sockets
Sep 28 06:49:15 dist-test-slave-twl6 krb5kdc[27756](info): commencing operation
krb5kdc: starting...
W20260928 06:49:16.700469 7360 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:49:16 dist-test-slave-twl6 krb5kdc[27756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:49:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:49:16.819171 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35261
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:16.929790 27769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:16.930086 27769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:16.930148 27769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:16.934145 27769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:16.934209 27769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:16.934229 27769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:16.934247 27769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:16.934262 27769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:16.939579 27769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35261
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:16.940749 27769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:16.941771 27769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:16.950182 27777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:16.950237 27769 server_base.cc:1034] running on GCE node
W20260928 06:49:16.950253 27774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:16.950344 27775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:16.950974 27769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:16.952186 27769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:16.953351 27769 hybrid_clock.cc:648] HybridClock initialized: now 1790578156953338 us; error 29 us; skew 500 ppm
Sep 28 06:49:16 dist-test-slave-twl6 krb5kdc[27756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:16.959647 27769 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:49:16.960645 27769 webserver.cc:460] Webserver started at http://127.0.0.1:41229/ using document root <none> and password file <none>
I20260928 06:49:16.961190 27769 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:16.961252 27769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:16.961422 27769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:16.963521 27769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "f9b8e29598234fefa35d747a9f8c4bca"
format_stamp: "Formatted at 2026-09-28 06:49:16 on dist-test-slave-twl6"
server_key: "193fdd579ef855b3da61fd6e8713f313"
server_key_iv: "661e8fb300013b5970b4dd51bb73d54a"
server_key_version: "encryptionkey@0"
I20260928 06:49:16.964134 27769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "f9b8e29598234fefa35d747a9f8c4bca"
format_stamp: "Formatted at 2026-09-28 06:49:16 on dist-test-slave-twl6"
server_key: "193fdd579ef855b3da61fd6e8713f313"
server_key_iv: "661e8fb300013b5970b4dd51bb73d54a"
server_key_version: "encryptionkey@0"
I20260928 06:49:16.968058 27769 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 06:49:16.970530 27784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:16.971829 27769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:49:16.972011 27769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "f9b8e29598234fefa35d747a9f8c4bca"
format_stamp: "Formatted at 2026-09-28 06:49:16 on dist-test-slave-twl6"
server_key: "193fdd579ef855b3da61fd6e8713f313"
server_key_iv: "661e8fb300013b5970b4dd51bb73d54a"
server_key_version: "encryptionkey@0"
I20260928 06:49:16.972124 27769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:16.982707 27769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:16.986352 27769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:16.986538 27769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:16.993842 27769 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35261
I20260928 06:49:16.993852 27836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35261 every 8 connection(s)
I20260928 06:49:16.994943 27769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:16.995725 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 27769
I20260928 06:49:16.995898 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:16.996225 7360 external_mini_cluster.cc:1428] Setting key 3315f77db4d27f99f04bd744ad39d939
I20260928 06:49:16.998539 27837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:17.004278 27837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b8e29598234fefa35d747a9f8c4bca: Bootstrap starting.
I20260928 06:49:17.005596 27840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:16.997645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50491 (local address 127.0.0.1:35261)
0928 06:49:16.998117 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:16.998133 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:16.998169 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:49:16.998887 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:16.998905 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:16.999263 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:16.999621 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:17.000072 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:17.001106 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:17.002034 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:17.002525 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:17.002711 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:17.003691 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:17.003696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:17.003710 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:17.004198 (+ 488us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:17.004258 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:17.004282 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:17.004434 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:17.004688 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 06:49:17.004857 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":158,"threads_started":1}
I20260928 06:49:17.006805 27837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9b8e29598234fefa35d747a9f8c4bca: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:17.007714 27837 log.cc:824] T 00000000000000000000000000000000 P f9b8e29598234fefa35d747a9f8c4bca: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:17.008914 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:49:17.009922 27837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b8e29598234fefa35d747a9f8c4bca: No bootstrap required, opened a new log
I20260928 06:49:17.013958 27837 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9b8e29598234fefa35d747a9f8c4bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b8e29598234fefa35d747a9f8c4bca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35261 } }
I20260928 06:49:17.014256 27837 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9b8e29598234fefa35d747a9f8c4bca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:17.014333 27837 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9b8e29598234fefa35d747a9f8c4bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b8e29598234fefa35d747a9f8c4bca, State: Initialized, Role: FOLLOWER
I20260928 06:49:17.014816 27837 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9b8e29598234fefa35d747a9f8c4bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b8e29598234fefa35d747a9f8c4bca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35261 } }
I20260928 06:49:17.014991 27837 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9b8e29598234fefa35d747a9f8c4bca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:49:17.015072 27837 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9b8e29598234fefa35d747a9f8c4bca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:49:17.015210 27837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9b8e29598234fefa35d747a9f8c4bca [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:17.015978 27840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:17.010936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37593 (local address 127.0.0.1:35261)
0928 06:49:17.011061 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:17.011070 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:17.011328 (+ 258us) server_negotiation.cc:407] Connection header received
0928 06:49:17.011449 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:17.011452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:17.011544 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:17.011649 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:17.011996 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:17.012861 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:17.014034 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:17.014275 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:17.014375 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:17.015406 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:17.015410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:17.015413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:17.015431 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:17.015465 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:17.015482 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:17.015593 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:17.015766 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 06:49:17.015805 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:49:17.016526 27837 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9b8e29598234fefa35d747a9f8c4bca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b8e29598234fefa35d747a9f8c4bca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35261 } }
I20260928 06:49:17.016902 27837 leader_election.cc:302] T 00000000000000000000000000000000 P f9b8e29598234fefa35d747a9f8c4bca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9b8e29598234fefa35d747a9f8c4bca; no voters:
I20260928 06:49:17.017306 27837 leader_election.cc:288] T 00000000000000000000000000000000 P f9b8e29598234fefa35d747a9f8c4bca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:49:17.017381 27847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9b8e29598234fefa35d747a9f8c4bca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:17.017663 27847 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9b8e29598234fefa35d747a9f8c4bca [term 1 LEADER]: Becoming Leader. State: Replica: f9b8e29598234fefa35d747a9f8c4bca, State: Running, Role: LEADER
I20260928 06:49:17.018078 27847 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9b8e29598234fefa35d747a9f8c4bca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b8e29598234fefa35d747a9f8c4bca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35261 } }
I20260928 06:49:17.018654 27837 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9b8e29598234fefa35d747a9f8c4bca [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:17.019869 27848 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b8e29598234fefa35d747a9f8c4bca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9b8e29598234fefa35d747a9f8c4bca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b8e29598234fefa35d747a9f8c4bca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35261 } } }
I20260928 06:49:17.019853 27849 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b8e29598234fefa35d747a9f8c4bca [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9b8e29598234fefa35d747a9f8c4bca. Latest consensus state: current_term: 1 leader_uuid: "f9b8e29598234fefa35d747a9f8c4bca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b8e29598234fefa35d747a9f8c4bca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35261 } } }
I20260928 06:49:17.020051 27848 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b8e29598234fefa35d747a9f8c4bca [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:17.020051 27849 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b8e29598234fefa35d747a9f8c4bca [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:17.020517 27856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:17.023898 27856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:49:17.030122 27856 catalog_manager.cc:1347] Generated new cluster ID: e545c21539944eb3bdc1d206bac53ae3
I20260928 06:49:17.030215 27856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:17.039517 27856 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:17.040853 27856 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:17.055255 27856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9b8e29598234fefa35d747a9f8c4bca: Generated new TSK 0
I20260928 06:49:17.056254 27856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:49:17.070784 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 27769
W20260928 06:49:17.076776 27754 connection.cc:535] client connection to 127.0.0.1:35261 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35261): Connection reset by peer (error 104)
2026-09-28T06:49:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:49:17 dist-test-slave-twl6 krb5kdc[27868](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:49:17 dist-test-slave-twl6 krb5kdc[27868](info): set up 2 sockets
Sep 28 06:49:17 dist-test-slave-twl6 krb5kdc[27868](info): commencing operation
krb5kdc: starting...
W20260928 06:49:18.100474 7360 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:49:18 dist-test-slave-twl6 krb5kdc[27868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:49:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:49:18.223527 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41151
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:18.334215 27881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:18.334506 27881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:18.334548 27881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:18.338490 27881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:18.338559 27881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:18.338578 27881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:18.338629 27881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:18.338660 27881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:18.343819 27881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41151
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:18.344988 27881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:18.346072 27881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:18.354472 27887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:18.354647 27881 server_base.cc:1034] running on GCE node
W20260928 06:49:18.354472 27889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:18.354579 27886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:18.355384 27881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:18.356611 27881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:18.357781 27881 hybrid_clock.cc:648] HybridClock initialized: now 1790578158357754 us; error 43 us; skew 500 ppm
Sep 28 06:49:18 dist-test-slave-twl6 krb5kdc[27868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:18.360563 27881 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:49:18.361455 27881 webserver.cc:460] Webserver started at http://127.0.0.1:41961/ using document root <none> and password file <none>
I20260928 06:49:18.362039 27881 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:18.362147 27881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:18.362393 27881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:18.364565 27881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "e05d47373cef4cd19d89c10413b20c5e"
format_stamp: "Formatted at 2026-09-28 06:49:18 on dist-test-slave-twl6"
server_key: "b18193c8187b233c057c5ad39bd85564"
server_key_iv: "c2d975f91761813332d8ab8604a3c78a"
server_key_version: "encryptionkey@0"
I20260928 06:49:18.365130 27881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "e05d47373cef4cd19d89c10413b20c5e"
format_stamp: "Formatted at 2026-09-28 06:49:18 on dist-test-slave-twl6"
server_key: "b18193c8187b233c057c5ad39bd85564"
server_key_iv: "c2d975f91761813332d8ab8604a3c78a"
server_key_version: "encryptionkey@0"
I20260928 06:49:18.369319 27881 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 06:49:18.371838 27896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:18.373119 27881 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:49:18.373262 27881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "e05d47373cef4cd19d89c10413b20c5e"
format_stamp: "Formatted at 2026-09-28 06:49:18 on dist-test-slave-twl6"
server_key: "b18193c8187b233c057c5ad39bd85564"
server_key_iv: "c2d975f91761813332d8ab8604a3c78a"
server_key_version: "encryptionkey@0"
I20260928 06:49:18.373380 27881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:18.377955 27881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:18.381193 27881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:18.381379 27881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:18.388284 27881 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41151
I20260928 06:49:18.388368 27948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41151 every 8 connection(s)
I20260928 06:49:18.389236 27881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:18.389954 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 27881
I20260928 06:49:18.390097 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:18.390455 7360 external_mini_cluster.cc:1428] Setting key 9babb9e2325109162f5670f9b1f27f4e
I20260928 06:49:18.393210 27949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:18.395738 27952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:18.391729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42099 (local address 127.0.0.1:41151)
0928 06:49:18.392195 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:18.392213 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:18.392256 (+ 43us) server_negotiation.cc:407] Connection header received
0928 06:49:18.392885 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:18.392901 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:18.393271 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:18.393636 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:18.394492 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:18.394496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:18.394513 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:18.394523 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:18.394582 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:18.394601 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:18.394720 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:18.394846 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 06:49:18.395024 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":209,"threads_started":1}
I20260928 06:49:18.398564 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:49:18.399398 27949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e05d47373cef4cd19d89c10413b20c5e: Bootstrap starting.
I20260928 06:49:18.402051 27949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e05d47373cef4cd19d89c10413b20c5e: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:18.402456 27952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:18.400854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49967 (local address 127.0.0.1:41151)
0928 06:49:18.400980 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:18.400984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:18.401145 (+ 161us) server_negotiation.cc:407] Connection header received
0928 06:49:18.401240 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:18.401243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:18.401278 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:18.401350 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:18.402076 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:18.402087 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:18.402090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:18.402106 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:18.402134 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:18.402148 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:18.402240 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:18.402311 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 06:49:18.402349 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 06:49:18.402900 27949 log.cc:824] T 00000000000000000000000000000000 P e05d47373cef4cd19d89c10413b20c5e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:18.404966 27949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e05d47373cef4cd19d89c10413b20c5e: No bootstrap required, opened a new log
I20260928 06:49:18.408599 27949 raft_consensus.cc:348] T 00000000000000000000000000000000 P e05d47373cef4cd19d89c10413b20c5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05d47373cef4cd19d89c10413b20c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41151 } }
I20260928 06:49:18.408823 27949 raft_consensus.cc:374] T 00000000000000000000000000000000 P e05d47373cef4cd19d89c10413b20c5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:18.408903 27949 raft_consensus.cc:729] T 00000000000000000000000000000000 P e05d47373cef4cd19d89c10413b20c5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e05d47373cef4cd19d89c10413b20c5e, State: Initialized, Role: FOLLOWER
I20260928 06:49:18.409392 27949 consensus_queue.cc:260] T 00000000000000000000000000000000 P e05d47373cef4cd19d89c10413b20c5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05d47373cef4cd19d89c10413b20c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41151 } }
I20260928 06:49:18.409574 27949 raft_consensus.cc:388] T 00000000000000000000000000000000 P e05d47373cef4cd19d89c10413b20c5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:49:18.409667 27949 raft_consensus.cc:482] T 00000000000000000000000000000000 P e05d47373cef4cd19d89c10413b20c5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:49:18.409787 27949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e05d47373cef4cd19d89c10413b20c5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:18.410738 27949 raft_consensus.cc:504] T 00000000000000000000000000000000 P e05d47373cef4cd19d89c10413b20c5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05d47373cef4cd19d89c10413b20c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41151 } }
I20260928 06:49:18.411058 27949 leader_election.cc:302] T 00000000000000000000000000000000 P e05d47373cef4cd19d89c10413b20c5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e05d47373cef4cd19d89c10413b20c5e; no voters:
I20260928 06:49:18.411365 27949 leader_election.cc:288] T 00000000000000000000000000000000 P e05d47373cef4cd19d89c10413b20c5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:49:18.411495 27959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e05d47373cef4cd19d89c10413b20c5e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:18.411720 27959 raft_consensus.cc:686] T 00000000000000000000000000000000 P e05d47373cef4cd19d89c10413b20c5e [term 1 LEADER]: Becoming Leader. State: Replica: e05d47373cef4cd19d89c10413b20c5e, State: Running, Role: LEADER
I20260928 06:49:18.412060 27959 consensus_queue.cc:237] T 00000000000000000000000000000000 P e05d47373cef4cd19d89c10413b20c5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05d47373cef4cd19d89c10413b20c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41151 } }
I20260928 06:49:18.412779 27949 sys_catalog.cc:564] T 00000000000000000000000000000000 P e05d47373cef4cd19d89c10413b20c5e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:18.413901 27960 sys_catalog.cc:455] T 00000000000000000000000000000000 P e05d47373cef4cd19d89c10413b20c5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e05d47373cef4cd19d89c10413b20c5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05d47373cef4cd19d89c10413b20c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41151 } } }
I20260928 06:49:18.414062 27960 sys_catalog.cc:458] T 00000000000000000000000000000000 P e05d47373cef4cd19d89c10413b20c5e [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:18.413899 27961 sys_catalog.cc:455] T 00000000000000000000000000000000 P e05d47373cef4cd19d89c10413b20c5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e05d47373cef4cd19d89c10413b20c5e. Latest consensus state: current_term: 1 leader_uuid: "e05d47373cef4cd19d89c10413b20c5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05d47373cef4cd19d89c10413b20c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41151 } } }
I20260928 06:49:18.414371 27961 sys_catalog.cc:458] T 00000000000000000000000000000000 P e05d47373cef4cd19d89c10413b20c5e [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:18.414556 27968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:18.417810 27968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:49:18.423386 27968 catalog_manager.cc:1347] Generated new cluster ID: 732e640012ea48c087d7179ebdfdad65
I20260928 06:49:18.423465 27968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:18.438349 27968 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:18.438659 27968 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:18.444343 27968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e05d47373cef4cd19d89c10413b20c5e: Generated new TSK 0
I20260928 06:49:18.445039 27968 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:49:18.450098 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 27881
2026-09-28T06:49:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1378 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:49:18 dist-test-slave-twl6 krb5kdc[27980](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:49:18 dist-test-slave-twl6 krb5kdc[27980](info): set up 2 sockets
Sep 28 06:49:18 dist-test-slave-twl6 krb5kdc[27980](info): commencing operation
krb5kdc: starting...
W20260928 06:49:19.452350 7360 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:49:19 dist-test-slave-twl6 krb5kdc[27980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:49:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:49:19.567571 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42103
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:19.672425 27993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:19.672672 27993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:19.672708 27993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:19.676697 27993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:19.676748 27993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:19.676766 27993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:19.676817 27993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:19.676847 27993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:19.681515 27993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:19.682461 27993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:19.683444 27993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:19.692060 27998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:19.692054 27999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:19.692165 27993 server_base.cc:1034] running on GCE node
W20260928 06:49:19.692054 28001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:19.692884 27993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:19.694000 27993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:19.695195 27993 hybrid_clock.cc:648] HybridClock initialized: now 1790578159695165 us; error 44 us; skew 500 ppm
Sep 28 06:49:19 dist-test-slave-twl6 krb5kdc[27980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:19.697955 27993 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:49:19.698846 27993 webserver.cc:460] Webserver started at http://127.0.0.1:37941/ using document root <none> and password file <none>
I20260928 06:49:19.699397 27993 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:19.699517 27993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:19.699751 27993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:19.701809 27993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "fdb3790009774ed380888bd17386a8b1"
format_stamp: "Formatted at 2026-09-28 06:49:19 on dist-test-slave-twl6"
server_key: "6a7f39c8c67a243d4f71b1e88b2cc77c"
server_key_iv: "a509c038c0e256a48003980393353534"
server_key_version: "encryptionkey@0"
I20260928 06:49:19.702381 27993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "fdb3790009774ed380888bd17386a8b1"
format_stamp: "Formatted at 2026-09-28 06:49:19 on dist-test-slave-twl6"
server_key: "6a7f39c8c67a243d4f71b1e88b2cc77c"
server_key_iv: "a509c038c0e256a48003980393353534"
server_key_version: "encryptionkey@0"
I20260928 06:49:19.706146 27993 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 06:49:19.708530 28008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:19.709714 27993 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:49:19.709822 27993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "fdb3790009774ed380888bd17386a8b1"
format_stamp: "Formatted at 2026-09-28 06:49:19 on dist-test-slave-twl6"
server_key: "6a7f39c8c67a243d4f71b1e88b2cc77c"
server_key_iv: "a509c038c0e256a48003980393353534"
server_key_version: "encryptionkey@0"
I20260928 06:49:19.709908 27993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:19.724381 27993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:19.727676 27993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:19.727830 27993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:19.734988 27993 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42103
I20260928 06:49:19.735057 28060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42103 every 8 connection(s)
I20260928 06:49:19.736014 27993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:19.738801 28061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:19.743362 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 27993
I20260928 06:49:19.743536 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:19.743834 7360 external_mini_cluster.cc:1428] Setting key 405513e2ec500e17655b9bc2a106ed56
I20260928 06:49:19.744565 28061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdb3790009774ed380888bd17386a8b1: Bootstrap starting.
I20260928 06:49:19.747152 28061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdb3790009774ed380888bd17386a8b1: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:19.748075 28061 log.cc:824] T 00000000000000000000000000000000 P fdb3790009774ed380888bd17386a8b1: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:19.750181 28061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdb3790009774ed380888bd17386a8b1: No bootstrap required, opened a new log
Sep 28 06:49:19 dist-test-slave-twl6 krb5kdc[27980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:49:19.753991 28061 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdb3790009774ed380888bd17386a8b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb3790009774ed380888bd17386a8b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42103 } }
I20260928 06:49:19.754278 28061 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdb3790009774ed380888bd17386a8b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:19.754364 28061 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdb3790009774ed380888bd17386a8b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdb3790009774ed380888bd17386a8b1, State: Initialized, Role: FOLLOWER
I20260928 06:49:19.754904 28061 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdb3790009774ed380888bd17386a8b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb3790009774ed380888bd17386a8b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42103 } }
I20260928 06:49:19.755084 28061 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdb3790009774ed380888bd17386a8b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:49:19.755154 28061 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdb3790009774ed380888bd17386a8b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:49:19.755275 28061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdb3790009774ed380888bd17386a8b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:19.756188 28061 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdb3790009774ed380888bd17386a8b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb3790009774ed380888bd17386a8b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42103 } }
I20260928 06:49:19.756531 28061 leader_election.cc:302] T 00000000000000000000000000000000 P fdb3790009774ed380888bd17386a8b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdb3790009774ed380888bd17386a8b1; no voters:
I20260928 06:49:19.756868 28061 leader_election.cc:288] T 00000000000000000000000000000000 P fdb3790009774ed380888bd17386a8b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:49:19.756935 28066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdb3790009774ed380888bd17386a8b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:19.757148 28066 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdb3790009774ed380888bd17386a8b1 [term 1 LEADER]: Becoming Leader. State: Replica: fdb3790009774ed380888bd17386a8b1, State: Running, Role: LEADER
I20260928 06:49:19.757452 28066 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdb3790009774ed380888bd17386a8b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb3790009774ed380888bd17386a8b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42103 } }
I20260928 06:49:19.758159 28061 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdb3790009774ed380888bd17386a8b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:19.759135 28067 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdb3790009774ed380888bd17386a8b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdb3790009774ed380888bd17386a8b1. Latest consensus state: current_term: 1 leader_uuid: "fdb3790009774ed380888bd17386a8b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb3790009774ed380888bd17386a8b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42103 } } }
I20260928 06:49:19.759181 28068 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdb3790009774ed380888bd17386a8b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdb3790009774ed380888bd17386a8b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb3790009774ed380888bd17386a8b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42103 } } }
I20260928 06:49:19.759343 28068 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdb3790009774ed380888bd17386a8b1 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:19.759372 28067 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdb3790009774ed380888bd17386a8b1 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:19.759756 28064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:19.745406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59991 (local address 127.0.0.1:42103)
0928 06:49:19.745854 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:19.745875 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:19.745918 (+ 43us) server_negotiation.cc:407] Connection header received
0928 06:49:19.746724 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:19.746747 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:19.747188 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:19.747567 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:19.748390 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:19.749777 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:19.750662 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:19.751332 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:19.751497 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:19.753972 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:19.753979 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:19.753995 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:19.754028 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:19.755999 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:19.756467 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:19.756473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:19.756485 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:19.756594 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:19.756892 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:19.756897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:19.756899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:19.757450 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:19.758198 (+ 748us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:19.758370 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 06:49:19.758673 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":135,"threads_started":1}
I20260928 06:49:19.762657 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 06:49:19.763717 28081 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fdb3790009774ed380888bd17386a8b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:49:19.763814 28081 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:49:19.763923 28075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:19.764817 28075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:49:19.770324 28075 catalog_manager.cc:1347] Generated new cluster ID: 0e5e29d5c5c94fb79c9839b76da14051
I20260928 06:49:19.770385 28075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:19.777099 28064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:19.768370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37531 (local address 127.0.0.1:42103)
0928 06:49:19.768486 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:19.768490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:19.768885 (+ 395us) server_negotiation.cc:407] Connection header received
0928 06:49:19.769022 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:19.769025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:19.769064 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:19.769155 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:19.769775 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:19.770495 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:19.771642 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:19.772051 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:19.772171 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:19.773689 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:19.773693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:19.773696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:19.773720 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:19.775408 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:19.775869 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:19.775923 (+ 54us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:19.775925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:19.776009 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:19.776276 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:19.776279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:19.776281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:19.776479 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:19.776601 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:19.776806 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 06:49:19.776916 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:49:19.788691 28075 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:19.789708 28075 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:19.795675 28075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdb3790009774ed380888bd17386a8b1: Generated new TSK 0
I20260928 06:49:19.796396 28075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:49:19.802479 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 27993
2026-09-28T06:49:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:49:19 dist-test-slave-twl6 krb5kdc[28092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:49:19 dist-test-slave-twl6 krb5kdc[28092](info): set up 2 sockets
Sep 28 06:49:19 dist-test-slave-twl6 krb5kdc[28092](info): commencing operation
krb5kdc: starting...
W20260928 06:49:20.772164 7360 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:49:20 dist-test-slave-twl6 krb5kdc[28092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:49:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:49:20.886529 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35033
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:20.988313 28105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:20.988557 28105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:20.988605 28105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:20.992383 28105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:20.992434 28105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:20.992452 28105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:20.992467 28105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:20.992527 28105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:20.997470 28105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:20.998456 28105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:20.999432 28105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:21.007592 28113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:21.007655 28105 server_base.cc:1034] running on GCE node
W20260928 06:49:21.007674 28111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:21.007594 28110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:21.008431 28105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:21.009498 28105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:21.010654 28105 hybrid_clock.cc:648] HybridClock initialized: now 1790578161010634 us; error 44 us; skew 500 ppm
Sep 28 06:49:21 dist-test-slave-twl6 krb5kdc[28092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:21.016353 28105 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:49:21.017292 28105 webserver.cc:460] Webserver started at http://127.0.0.1:45915/ using document root <none> and password file <none>
I20260928 06:49:21.017845 28105 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:21.017906 28105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:21.018159 28105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:21.020151 28105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "e61e07070cbc415c9c0160cde223effb"
format_stamp: "Formatted at 2026-09-28 06:49:21 on dist-test-slave-twl6"
server_key: "07021c3befc473bd9969a3f530956a26"
server_key_iv: "f5cc5f89f531645f60318f96fa9a3aa2"
server_key_version: "encryptionkey@0"
I20260928 06:49:21.020699 28105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "e61e07070cbc415c9c0160cde223effb"
format_stamp: "Formatted at 2026-09-28 06:49:21 on dist-test-slave-twl6"
server_key: "07021c3befc473bd9969a3f530956a26"
server_key_iv: "f5cc5f89f531645f60318f96fa9a3aa2"
server_key_version: "encryptionkey@0"
I20260928 06:49:21.024430 28105 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:49:21.027227 28120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:21.028445 28105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:49:21.028584 28105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "e61e07070cbc415c9c0160cde223effb"
format_stamp: "Formatted at 2026-09-28 06:49:21 on dist-test-slave-twl6"
server_key: "07021c3befc473bd9969a3f530956a26"
server_key_iv: "f5cc5f89f531645f60318f96fa9a3aa2"
server_key_version: "encryptionkey@0"
I20260928 06:49:21.028692 28105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:21.046231 28105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:21.049145 28105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:21.049341 28105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:21.056293 28105 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35033
I20260928 06:49:21.056318 28172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35033 every 8 connection(s)
I20260928 06:49:21.057181 28105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:21.059660 28173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:21.062987 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 28105
I20260928 06:49:21.063149 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:21.063458 7360 external_mini_cluster.cc:1428] Setting key 2d283611c5ee5997b34389df1abf400c
I20260928 06:49:21.065439 28173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e61e07070cbc415c9c0160cde223effb: Bootstrap starting.
I20260928 06:49:21.067510 28173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e61e07070cbc415c9c0160cde223effb: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:21.068264 28173 log.cc:824] T 00000000000000000000000000000000 P e61e07070cbc415c9c0160cde223effb: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:21.070027 28173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e61e07070cbc415c9c0160cde223effb: No bootstrap required, opened a new log
I20260928 06:49:21.072386 28176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:21.064942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53935 (local address 127.0.0.1:35033)
0928 06:49:21.065408 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:21.065451 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:21.065485 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:49:21.066120 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:21.066135 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:21.066496 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:21.066828 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:21.067228 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:21.068379 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:21.069230 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:21.069820 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:21.069984 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:21.070786 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:21.070792 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:21.070811 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:21.071156 (+ 345us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:21.071208 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:21.071228 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:21.071332 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:21.071470 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 06:49:21.071666 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":121,"threads_started":1}
I20260928 06:49:21.073467 28173 raft_consensus.cc:348] T 00000000000000000000000000000000 P e61e07070cbc415c9c0160cde223effb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61e07070cbc415c9c0160cde223effb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35033 } }
I20260928 06:49:21.073701 28173 raft_consensus.cc:374] T 00000000000000000000000000000000 P e61e07070cbc415c9c0160cde223effb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:21.073777 28173 raft_consensus.cc:729] T 00000000000000000000000000000000 P e61e07070cbc415c9c0160cde223effb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e61e07070cbc415c9c0160cde223effb, State: Initialized, Role: FOLLOWER
I20260928 06:49:21.074249 28173 consensus_queue.cc:260] T 00000000000000000000000000000000 P e61e07070cbc415c9c0160cde223effb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61e07070cbc415c9c0160cde223effb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35033 } }
I20260928 06:49:21.074399 28173 raft_consensus.cc:388] T 00000000000000000000000000000000 P e61e07070cbc415c9c0160cde223effb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:49:21.074471 28173 raft_consensus.cc:482] T 00000000000000000000000000000000 P e61e07070cbc415c9c0160cde223effb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:49:21.074592 28173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e61e07070cbc415c9c0160cde223effb [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:21.075107 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:49:21.075542 28173 raft_consensus.cc:504] T 00000000000000000000000000000000 P e61e07070cbc415c9c0160cde223effb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61e07070cbc415c9c0160cde223effb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35033 } }
I20260928 06:49:21.075851 28173 leader_election.cc:302] T 00000000000000000000000000000000 P e61e07070cbc415c9c0160cde223effb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e61e07070cbc415c9c0160cde223effb; no voters:
I20260928 06:49:21.076201 28173 leader_election.cc:288] T 00000000000000000000000000000000 P e61e07070cbc415c9c0160cde223effb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:49:21.076328 28178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e61e07070cbc415c9c0160cde223effb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:21.076537 28178 raft_consensus.cc:686] T 00000000000000000000000000000000 P e61e07070cbc415c9c0160cde223effb [term 1 LEADER]: Becoming Leader. State: Replica: e61e07070cbc415c9c0160cde223effb, State: Running, Role: LEADER
I20260928 06:49:21.076916 28178 consensus_queue.cc:237] T 00000000000000000000000000000000 P e61e07070cbc415c9c0160cde223effb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61e07070cbc415c9c0160cde223effb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35033 } }
I20260928 06:49:21.077467 28173 sys_catalog.cc:564] T 00000000000000000000000000000000 P e61e07070cbc415c9c0160cde223effb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:21.078500 28184 sys_catalog.cc:455] T 00000000000000000000000000000000 P e61e07070cbc415c9c0160cde223effb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e61e07070cbc415c9c0160cde223effb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61e07070cbc415c9c0160cde223effb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35033 } } }
I20260928 06:49:21.078629 28184 sys_catalog.cc:458] T 00000000000000000000000000000000 P e61e07070cbc415c9c0160cde223effb [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:21.078930 28182 sys_catalog.cc:455] T 00000000000000000000000000000000 P e61e07070cbc415c9c0160cde223effb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e61e07070cbc415c9c0160cde223effb. Latest consensus state: current_term: 1 leader_uuid: "e61e07070cbc415c9c0160cde223effb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61e07070cbc415c9c0160cde223effb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35033 } } }
I20260928 06:49:21.079075 28182 sys_catalog.cc:458] T 00000000000000000000000000000000 P e61e07070cbc415c9c0160cde223effb [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:21.081409 28176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:21.077467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37483 (local address 127.0.0.1:35033)
0928 06:49:21.077566 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:21.077570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:21.077583 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:49:21.077678 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:21.077682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:21.077721 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:21.077765 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:21.078199 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:21.078914 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:21.079971 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:21.080192 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:21.080295 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:21.081043 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:21.081046 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:21.081049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:21.081059 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:21.081084 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:21.081095 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:21.081164 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:21.081260 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 06:49:21.081304 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 06:49:21.082203 28198 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e61e07070cbc415c9c0160cde223effb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:49:21.082268 28198 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:49:21.082329 28192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:21.082903 28192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:49:21.087590 28192 catalog_manager.cc:1347] Generated new cluster ID: 155f435358f34efd8d77863004ff28ea
I20260928 06:49:21.087661 28192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:21.108176 28192 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:21.109045 28192 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:21.114843 28192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e61e07070cbc415c9c0160cde223effb: Generated new TSK 0
I20260928 06:49:21.115540 28192 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:49:21.126636 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 28105
2026-09-28T06:49:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:49:21 dist-test-slave-twl6 krb5kdc[28204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:49:21 dist-test-slave-twl6 krb5kdc[28204](info): set up 2 sockets
Sep 28 06:49:21 dist-test-slave-twl6 krb5kdc[28204](info): commencing operation
krb5kdc: starting...
W20260928 06:49:22.112280 7360 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:49:22 dist-test-slave-twl6 krb5kdc[28204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:49:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:49:22.224893 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42529
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:22.330369 28217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:22.330747 28217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:22.330821 28217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:22.334769 28217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:22.334857 28217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:22.334897 28217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:22.334955 28217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:22.334995 28217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:22.340080 28217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:22.341111 28217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:22.342173 28217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:22.350572 28225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:22.350553 28223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:22.350621 28222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:22.350960 28217 server_base.cc:1034] running on GCE node
I20260928 06:49:22.351533 28217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:22.352653 28217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:22.353834 28217 hybrid_clock.cc:648] HybridClock initialized: now 1790578162353822 us; error 24 us; skew 500 ppm
Sep 28 06:49:22 dist-test-slave-twl6 krb5kdc[28204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:22.360975 28217 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:49:22.361832 28217 webserver.cc:460] Webserver started at http://127.0.0.1:34013/ using document root <none> and password file <none>
I20260928 06:49:22.362368 28217 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:22.362424 28217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:22.362583 28217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:22.364758 28217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "514bdc65d95a401697104775778ea071"
format_stamp: "Formatted at 2026-09-28 06:49:22 on dist-test-slave-twl6"
server_key: "3826c65175e63ad3f6259cf7facb31fd"
server_key_iv: "1fe6ca577f76bc0370aabf599c4d867e"
server_key_version: "encryptionkey@0"
I20260928 06:49:22.365267 28217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "514bdc65d95a401697104775778ea071"
format_stamp: "Formatted at 2026-09-28 06:49:22 on dist-test-slave-twl6"
server_key: "3826c65175e63ad3f6259cf7facb31fd"
server_key_iv: "1fe6ca577f76bc0370aabf599c4d867e"
server_key_version: "encryptionkey@0"
I20260928 06:49:22.368944 28217 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 06:49:22.371419 28232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:22.372666 28217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:49:22.372799 28217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "514bdc65d95a401697104775778ea071"
format_stamp: "Formatted at 2026-09-28 06:49:22 on dist-test-slave-twl6"
server_key: "3826c65175e63ad3f6259cf7facb31fd"
server_key_iv: "1fe6ca577f76bc0370aabf599c4d867e"
server_key_version: "encryptionkey@0"
I20260928 06:49:22.372911 28217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:22.377509 28217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:22.380424 28217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:22.380615 28217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:22.387576 28217 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42529
I20260928 06:49:22.387604 28284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42529 every 8 connection(s)
I20260928 06:49:22.388577 28217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:22.391188 28285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:22.391306 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 28217
I20260928 06:49:22.391454 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:22.391742 7360 external_mini_cluster.cc:1428] Setting key 120cec7b5fcc10f9dc0fb6ddd0e11bd7
I20260928 06:49:22.397378 28288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:22.393184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46819 (local address 127.0.0.1:42529)
0928 06:49:22.393640 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:22.393659 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:22.393703 (+ 44us) server_negotiation.cc:407] Connection header received
0928 06:49:22.394500 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:22.394525 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:22.394896 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:22.395273 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:22.396019 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:22.396024 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:22.396038 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:22.396051 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:22.396121 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:22.396148 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:22.396286 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:22.396355 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 06:49:22.396503 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":138,"threads_started":1}
I20260928 06:49:22.398268 28285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 514bdc65d95a401697104775778ea071: Bootstrap starting.
I20260928 06:49:22.400281 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:49:22.401000 28285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 514bdc65d95a401697104775778ea071: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:22.401901 28285 log.cc:824] T 00000000000000000000000000000000 P 514bdc65d95a401697104775778ea071: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:22.403707 28285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 514bdc65d95a401697104775778ea071: No bootstrap required, opened a new log
I20260928 06:49:22.405424 28288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:22.402474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44377 (local address 127.0.0.1:42529)
0928 06:49:22.402603 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:22.402607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:22.404070 (+ 1463us) server_negotiation.cc:407] Connection header received
0928 06:49:22.404210 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:22.404213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:22.404250 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:22.404333 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:22.405046 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:22.405049 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:22.405052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:22.405070 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:22.405092 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:22.405101 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:22.405185 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:22.405273 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 06:49:22.405310 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 06:49:22.407627 28285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 514bdc65d95a401697104775778ea071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "514bdc65d95a401697104775778ea071" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42529 } }
I20260928 06:49:22.407840 28285 raft_consensus.cc:374] T 00000000000000000000000000000000 P 514bdc65d95a401697104775778ea071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:22.407946 28285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 514bdc65d95a401697104775778ea071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 514bdc65d95a401697104775778ea071, State: Initialized, Role: FOLLOWER
I20260928 06:49:22.408356 28285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 514bdc65d95a401697104775778ea071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "514bdc65d95a401697104775778ea071" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42529 } }
I20260928 06:49:22.408496 28285 raft_consensus.cc:388] T 00000000000000000000000000000000 P 514bdc65d95a401697104775778ea071 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:49:22.408561 28285 raft_consensus.cc:482] T 00000000000000000000000000000000 P 514bdc65d95a401697104775778ea071 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:49:22.408668 28285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 514bdc65d95a401697104775778ea071 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:22.409466 28285 raft_consensus.cc:504] T 00000000000000000000000000000000 P 514bdc65d95a401697104775778ea071 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "514bdc65d95a401697104775778ea071" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42529 } }
I20260928 06:49:22.409795 28285 leader_election.cc:302] T 00000000000000000000000000000000 P 514bdc65d95a401697104775778ea071 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 514bdc65d95a401697104775778ea071; no voters:
I20260928 06:49:22.410087 28285 leader_election.cc:288] T 00000000000000000000000000000000 P 514bdc65d95a401697104775778ea071 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:49:22.410167 28295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 514bdc65d95a401697104775778ea071 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:22.410465 28295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 514bdc65d95a401697104775778ea071 [term 1 LEADER]: Becoming Leader. State: Replica: 514bdc65d95a401697104775778ea071, State: Running, Role: LEADER
I20260928 06:49:22.410789 28295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 514bdc65d95a401697104775778ea071 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "514bdc65d95a401697104775778ea071" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42529 } }
I20260928 06:49:22.411358 28285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 514bdc65d95a401697104775778ea071 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:22.412323 28297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 514bdc65d95a401697104775778ea071 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 514bdc65d95a401697104775778ea071. Latest consensus state: current_term: 1 leader_uuid: "514bdc65d95a401697104775778ea071" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "514bdc65d95a401697104775778ea071" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42529 } } }
I20260928 06:49:22.412452 28297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 514bdc65d95a401697104775778ea071 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:22.412768 28296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 514bdc65d95a401697104775778ea071 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "514bdc65d95a401697104775778ea071" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "514bdc65d95a401697104775778ea071" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42529 } } }
I20260928 06:49:22.412819 28304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:22.412850 28296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 514bdc65d95a401697104775778ea071 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:22.415395 28304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:49:22.420714 28304 catalog_manager.cc:1347] Generated new cluster ID: b2dbc943c78a446db9907cc5f8e9b68b
I20260928 06:49:22.420784 28304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:22.434950 28304 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:22.435261 28304 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:22.446985 28304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 514bdc65d95a401697104775778ea071: Generated new TSK 0
I20260928 06:49:22.447714 28304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:49:22.453109 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 28217
2026-09-28T06:49:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:49:22 dist-test-slave-twl6 krb5kdc[28316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:49:22 dist-test-slave-twl6 krb5kdc[28316](info): set up 2 sockets
Sep 28 06:49:22 dist-test-slave-twl6 krb5kdc[28316](info): commencing operation
krb5kdc: starting...
W20260928 06:49:23.460377 7360 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:49:23 dist-test-slave-twl6 krb5kdc[28316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:49:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:49:23.578292 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42465
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:23.684984 28329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:23.685284 28329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:23.685356 28329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:23.689340 28329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:23.689429 28329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:23.689469 28329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:23.689529 28329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:23.689569 28329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:23.694651 28329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:23.695706 28329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:23.696731 28329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:23.705232 28337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:23.705232 28334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:23.705351 28329 server_base.cc:1034] running on GCE node
W20260928 06:49:23.705228 28335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:23.706226 28329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:23.707283 28329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:23.708467 28329 hybrid_clock.cc:648] HybridClock initialized: now 1790578163708454 us; error 26 us; skew 500 ppm
Sep 28 06:49:23 dist-test-slave-twl6 krb5kdc[28316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:23.711100 28329 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:49:23.712045 28329 webserver.cc:460] Webserver started at http://127.0.0.1:32875/ using document root <none> and password file <none>
I20260928 06:49:23.712601 28329 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:23.712690 28329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:23.712914 28329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:23.714982 28329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "46642671e19d43a48d04d09555ecfa51"
format_stamp: "Formatted at 2026-09-28 06:49:23 on dist-test-slave-twl6"
server_key: "a400377d8c83456f0a8567d451540d39"
server_key_iv: "8560ee4f255b80e92ef916880c391041"
server_key_version: "encryptionkey@0"
I20260928 06:49:23.715566 28329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "46642671e19d43a48d04d09555ecfa51"
format_stamp: "Formatted at 2026-09-28 06:49:23 on dist-test-slave-twl6"
server_key: "a400377d8c83456f0a8567d451540d39"
server_key_iv: "8560ee4f255b80e92ef916880c391041"
server_key_version: "encryptionkey@0"
I20260928 06:49:23.719297 28329 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 06:49:23.721824 28344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:23.723018 28329 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:49:23.723158 28329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "46642671e19d43a48d04d09555ecfa51"
format_stamp: "Formatted at 2026-09-28 06:49:23 on dist-test-slave-twl6"
server_key: "a400377d8c83456f0a8567d451540d39"
server_key_iv: "8560ee4f255b80e92ef916880c391041"
server_key_version: "encryptionkey@0"
I20260928 06:49:23.723272 28329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:23.741703 28329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:23.744891 28329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:23.745105 28329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:23.752467 28329 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42465
I20260928 06:49:23.752492 28396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42465 every 8 connection(s)
I20260928 06:49:23.753485 28329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:23.754014 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 28329
I20260928 06:49:23.754187 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:23.754511 7360 external_mini_cluster.cc:1428] Setting key 8e2a1d57a6a96f4520af4dfe7b7e2713
I20260928 06:49:23.756754 28397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:23.762274 28397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46642671e19d43a48d04d09555ecfa51: Bootstrap starting.
Sep 28 06:49:23 dist-test-slave-twl6 krb5kdc[28316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:49:23.764689 28397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46642671e19d43a48d04d09555ecfa51: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:23.765499 28397 log.cc:824] T 00000000000000000000000000000000 P 46642671e19d43a48d04d09555ecfa51: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:23.767721 28397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46642671e19d43a48d04d09555ecfa51: No bootstrap required, opened a new log
I20260928 06:49:23.768684 28400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:23.755973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:33665 (local address 127.0.0.1:42465)
0928 06:49:23.756442 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:23.756472 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:23.756528 (+ 56us) server_negotiation.cc:407] Connection header received
0928 06:49:23.757233 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:23.757252 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:23.757648 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:23.758004 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:23.758808 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:23.759771 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:23.760636 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:23.761083 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:23.761252 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:23.763832 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:23.763837 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:23.763849 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:23.763921 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:23.765809 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:23.766343 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:23.766348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:23.766353 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:23.766461 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:23.766811 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:23.766813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:23.766814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:23.767200 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:23.767360 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:23.767685 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 06:49:23.767974 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":165,"threads_started":1}
I20260928 06:49:23.770750 28397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46642671e19d43a48d04d09555ecfa51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46642671e19d43a48d04d09555ecfa51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42465 } }
I20260928 06:49:23.771037 28397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46642671e19d43a48d04d09555ecfa51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:23.771127 28397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46642671e19d43a48d04d09555ecfa51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46642671e19d43a48d04d09555ecfa51, State: Initialized, Role: FOLLOWER
I20260928 06:49:23.771618 28397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46642671e19d43a48d04d09555ecfa51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46642671e19d43a48d04d09555ecfa51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42465 } }
I20260928 06:49:23.771746 28397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46642671e19d43a48d04d09555ecfa51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:49:23.771847 28397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46642671e19d43a48d04d09555ecfa51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:49:23.772035 28397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46642671e19d43a48d04d09555ecfa51 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:23.772166 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:49:23.773036 28397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46642671e19d43a48d04d09555ecfa51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46642671e19d43a48d04d09555ecfa51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42465 } }
I20260928 06:49:23.773401 28397 leader_election.cc:302] T 00000000000000000000000000000000 P 46642671e19d43a48d04d09555ecfa51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46642671e19d43a48d04d09555ecfa51; no voters:
I20260928 06:49:23.773739 28397 leader_election.cc:288] T 00000000000000000000000000000000 P 46642671e19d43a48d04d09555ecfa51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:49:23.773864 28406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46642671e19d43a48d04d09555ecfa51 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:23.774134 28406 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46642671e19d43a48d04d09555ecfa51 [term 1 LEADER]: Becoming Leader. State: Replica: 46642671e19d43a48d04d09555ecfa51, State: Running, Role: LEADER
I20260928 06:49:23.774417 28406 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46642671e19d43a48d04d09555ecfa51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46642671e19d43a48d04d09555ecfa51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42465 } }
I20260928 06:49:23.775106 28397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46642671e19d43a48d04d09555ecfa51 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:23.776255 28407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46642671e19d43a48d04d09555ecfa51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46642671e19d43a48d04d09555ecfa51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46642671e19d43a48d04d09555ecfa51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42465 } } }
I20260928 06:49:23.776382 28407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46642671e19d43a48d04d09555ecfa51 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:23.776718 28408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46642671e19d43a48d04d09555ecfa51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46642671e19d43a48d04d09555ecfa51. Latest consensus state: current_term: 1 leader_uuid: "46642671e19d43a48d04d09555ecfa51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46642671e19d43a48d04d09555ecfa51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42465 } } }
I20260928 06:49:23.776779 28416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:23.776875 28408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46642671e19d43a48d04d09555ecfa51 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:23.779707 28416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:49:23.785750 28400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:23.774587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:41609 (local address 127.0.0.1:42465)
0928 06:49:23.774688 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:23.774692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:23.774707 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:49:23.774743 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:23.774746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:23.774791 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:23.774873 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:23.777725 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:23.778453 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:23.779663 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:23.780035 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:23.780135 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:23.782044 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:23.782048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:23.782050 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:23.782075 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:23.783750 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:23.784294 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:23.784297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:23.784299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:23.784390 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:23.784826 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:23.784829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:23.784831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:23.785047 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:23.785191 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:23.785495 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 06:49:23.785603 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:49:23.786132 28416 catalog_manager.cc:1347] Generated new cluster ID: a7affe57e3cd4292bb8b2bee8f14e2ec
I20260928 06:49:23.786211 28416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:23.808248 28416 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:23.809237 28416 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:23.818095 28416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46642671e19d43a48d04d09555ecfa51: Generated new TSK 0
I20260928 06:49:23.818867 28416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:49:23.823989 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 28329
W20260928 06:49:23.829061 28314 connection.cc:535] client connection to 127.0.0.1:42465 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42465): Connection reset by peer (error 104)
2026-09-28T06:49:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1370 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:49:23 dist-test-slave-twl6 krb5kdc[28428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:49:23 dist-test-slave-twl6 krb5kdc[28428](info): set up 2 sockets
Sep 28 06:49:23 dist-test-slave-twl6 krb5kdc[28428](info): commencing operation
krb5kdc: starting...
W20260928 06:49:24.836575 7360 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:49:24 dist-test-slave-twl6 krb5kdc[28428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:49:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:49:24.961254 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40463
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:25.071322 28441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:25.071605 28441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:25.071648 28441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:25.075598 28441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:25.075668 28441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:25.075687 28441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:25.075704 28441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:25.075721 28441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:25.080891 28441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:25.081979 28441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:25.082942 28441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:25.091615 28449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:25.091615 28447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:25.091756 28441 server_base.cc:1034] running on GCE node
W20260928 06:49:25.091676 28446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:25.092654 28441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:25.093854 28441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:25.095021 28441 hybrid_clock.cc:648] HybridClock initialized: now 1790578165095004 us; error 34 us; skew 500 ppm
Sep 28 06:49:25 dist-test-slave-twl6 krb5kdc[28428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:25.097994 28441 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:49:25.098847 28441 webserver.cc:460] Webserver started at http://127.0.0.1:34923/ using document root <none> and password file <none>
I20260928 06:49:25.099360 28441 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:25.099418 28441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:25.099586 28441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:25.101670 28441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "a7124eadf82d4890a1cc6c343594ab21"
format_stamp: "Formatted at 2026-09-28 06:49:25 on dist-test-slave-twl6"
server_key: "f88cf97911d9e12c11808f1300de6df6"
server_key_iv: "5a895c96f5f6e7c3ce502e1519e43625"
server_key_version: "encryptionkey@0"
I20260928 06:49:25.102196 28441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "a7124eadf82d4890a1cc6c343594ab21"
format_stamp: "Formatted at 2026-09-28 06:49:25 on dist-test-slave-twl6"
server_key: "f88cf97911d9e12c11808f1300de6df6"
server_key_iv: "5a895c96f5f6e7c3ce502e1519e43625"
server_key_version: "encryptionkey@0"
I20260928 06:49:25.106086 28441 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260928 06:49:25.108633 28456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:25.109838 28441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:49:25.110010 28441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "a7124eadf82d4890a1cc6c343594ab21"
format_stamp: "Formatted at 2026-09-28 06:49:25 on dist-test-slave-twl6"
server_key: "f88cf97911d9e12c11808f1300de6df6"
server_key_iv: "5a895c96f5f6e7c3ce502e1519e43625"
server_key_version: "encryptionkey@0"
I20260928 06:49:25.110133 28441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:25.121686 28441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:25.124577 28441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:25.124785 28441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:25.131940 28441 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40463
I20260928 06:49:25.132005 28508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40463 every 8 connection(s)
I20260928 06:49:25.132939 28441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:25.135788 28509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:25.137485 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 28441
I20260928 06:49:25.137665 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:25.137949 7360 external_mini_cluster.cc:1428] Setting key d2a6d3533bf3cb063baaa5392af447dc
I20260928 06:49:25.142120 28509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7124eadf82d4890a1cc6c343594ab21: Bootstrap starting.
I20260928 06:49:25.144537 28509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7124eadf82d4890a1cc6c343594ab21: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:25.145272 28509 log.cc:824] T 00000000000000000000000000000000 P a7124eadf82d4890a1cc6c343594ab21: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:25.147419 28512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:25.139397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:56993 (local address 127.0.0.1:40463)
0928 06:49:25.139838 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:25.139853 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:25.139939 (+ 86us) server_negotiation.cc:407] Connection header received
0928 06:49:25.140582 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:25.140598 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:25.140947 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:25.141287 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:25.141682 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:25.142685 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:25.143603 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:25.144158 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:25.144332 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:25.145270 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:25.145276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:25.145289 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:25.145803 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:25.145886 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:25.145915 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:25.146064 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:25.146380 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 06:49:25.146520 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":127,"threads_started":1}
I20260928 06:49:25.147759 28509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7124eadf82d4890a1cc6c343594ab21: No bootstrap required, opened a new log
I20260928 06:49:25.150833 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:49:25.152169 28509 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7124eadf82d4890a1cc6c343594ab21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7124eadf82d4890a1cc6c343594ab21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40463 } }
I20260928 06:49:25.152479 28509 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7124eadf82d4890a1cc6c343594ab21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:25.152544 28509 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7124eadf82d4890a1cc6c343594ab21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7124eadf82d4890a1cc6c343594ab21, State: Initialized, Role: FOLLOWER
I20260928 06:49:25.153000 28509 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7124eadf82d4890a1cc6c343594ab21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7124eadf82d4890a1cc6c343594ab21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40463 } }
I20260928 06:49:25.153182 28509 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7124eadf82d4890a1cc6c343594ab21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:49:25.153297 28509 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7124eadf82d4890a1cc6c343594ab21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:49:25.153460 28509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7124eadf82d4890a1cc6c343594ab21 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:25.154428 28509 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7124eadf82d4890a1cc6c343594ab21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7124eadf82d4890a1cc6c343594ab21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40463 } }
I20260928 06:49:25.154776 28509 leader_election.cc:302] T 00000000000000000000000000000000 P a7124eadf82d4890a1cc6c343594ab21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7124eadf82d4890a1cc6c343594ab21; no voters:
I20260928 06:49:25.155120 28509 leader_election.cc:288] T 00000000000000000000000000000000 P a7124eadf82d4890a1cc6c343594ab21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:49:25.155215 28519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7124eadf82d4890a1cc6c343594ab21 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:25.155408 28519 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7124eadf82d4890a1cc6c343594ab21 [term 1 LEADER]: Becoming Leader. State: Replica: a7124eadf82d4890a1cc6c343594ab21, State: Running, Role: LEADER
I20260928 06:49:25.155767 28519 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7124eadf82d4890a1cc6c343594ab21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7124eadf82d4890a1cc6c343594ab21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40463 } }
I20260928 06:49:25.156476 28509 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7124eadf82d4890a1cc6c343594ab21 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:25.157473 28520 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7124eadf82d4890a1cc6c343594ab21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7124eadf82d4890a1cc6c343594ab21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7124eadf82d4890a1cc6c343594ab21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40463 } } }
I20260928 06:49:25.157874 28512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:25.153143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:40549 (local address 127.0.0.1:40463)
0928 06:49:25.153267 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:25.153271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:25.153607 (+ 336us) server_negotiation.cc:407] Connection header received
0928 06:49:25.153778 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:25.153781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:25.153833 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:25.153914 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:25.154185 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:25.154782 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:25.155849 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:25.156159 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:25.156280 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:25.157205 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:25.157214 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:25.157217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:25.157231 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:25.157259 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:25.157273 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:25.157340 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:25.157618 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 06:49:25.157660 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:49:25.157667 28520 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7124eadf82d4890a1cc6c343594ab21 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:25.158671 28520 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7124eadf82d4890a1cc6c343594ab21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7124eadf82d4890a1cc6c343594ab21. Latest consensus state: current_term: 1 leader_uuid: "a7124eadf82d4890a1cc6c343594ab21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7124eadf82d4890a1cc6c343594ab21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40463 } } }
I20260928 06:49:25.158768 28520 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7124eadf82d4890a1cc6c343594ab21 [sys.catalog]: This master's current role is: LEADER
W20260928 06:49:25.161821 28535 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7124eadf82d4890a1cc6c343594ab21: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:49:25.161909 28535 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:49:25.162024 28532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:25.162725 28532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:49:25.168413 28532 catalog_manager.cc:1347] Generated new cluster ID: 727ea75b743b42138e773552a9958ce4
I20260928 06:49:25.168488 28532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:25.181648 28532 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:25.182958 28532 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:25.189306 28532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7124eadf82d4890a1cc6c343594ab21: Generated new TSK 0
I20260928 06:49:25.190135 28532 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:49:25.194530 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 28441
W20260928 06:49:25.200088 28426 connection.cc:535] client connection to 127.0.0.1:40463 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40463): Connection reset by peer (error 104)
2026-09-28T06:49:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:49:25 dist-test-slave-twl6 krb5kdc[28541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:49:25 dist-test-slave-twl6 krb5kdc[28541](info): set up 2 sockets
Sep 28 06:49:25 dist-test-slave-twl6 krb5kdc[28541](info): commencing operation
krb5kdc: starting...
W20260928 06:49:26.220371 7360 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:49:26 dist-test-slave-twl6 krb5kdc[28541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:49:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:49:26.343858 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39727
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:26.455793 28554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:26.456094 28554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:26.456167 28554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:26.460323 28554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:26.460391 28554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:26.460412 28554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:26.460464 28554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:26.460495 28554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:26.465750 28554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:26.466874 28554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:26.467980 28554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:26.477030 28559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:26.477001 28560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:26.477067 28562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:26.477324 28554 server_base.cc:1034] running on GCE node
I20260928 06:49:26.478053 28554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:26.479246 28554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:26.480432 28554 hybrid_clock.cc:648] HybridClock initialized: now 1790578166480400 us; error 48 us; skew 500 ppm
Sep 28 06:49:26 dist-test-slave-twl6 krb5kdc[28541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:26.483361 28554 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:49:26.484412 28554 webserver.cc:460] Webserver started at http://127.0.0.1:34653/ using document root <none> and password file <none>
I20260928 06:49:26.485013 28554 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:26.485113 28554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:26.485356 28554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:26.487515 28554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "3dc9ef1e37164a1abb28c63ea6fa614b"
format_stamp: "Formatted at 2026-09-28 06:49:26 on dist-test-slave-twl6"
server_key: "6c1ed37969e1eb1b5baed59498cf4072"
server_key_iv: "0b16224608bd0b7bae75ae064cb32bf7"
server_key_version: "encryptionkey@0"
I20260928 06:49:26.488157 28554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "3dc9ef1e37164a1abb28c63ea6fa614b"
format_stamp: "Formatted at 2026-09-28 06:49:26 on dist-test-slave-twl6"
server_key: "6c1ed37969e1eb1b5baed59498cf4072"
server_key_iv: "0b16224608bd0b7bae75ae064cb32bf7"
server_key_version: "encryptionkey@0"
I20260928 06:49:26.492199 28554 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 06:49:26.494721 28569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:26.495972 28554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:49:26.496109 28554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "3dc9ef1e37164a1abb28c63ea6fa614b"
format_stamp: "Formatted at 2026-09-28 06:49:26 on dist-test-slave-twl6"
server_key: "6c1ed37969e1eb1b5baed59498cf4072"
server_key_iv: "0b16224608bd0b7bae75ae064cb32bf7"
server_key_version: "encryptionkey@0"
I20260928 06:49:26.496251 28554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:26.500912 28554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:26.503811 28554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:26.504093 28554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:26.510972 28554 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39727
I20260928 06:49:26.511060 28621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39727 every 8 connection(s)
I20260928 06:49:26.512028 28554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:26.514853 28622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:26.520511 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 28554
I20260928 06:49:26.520690 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:26.521039 7360 external_mini_cluster.cc:1428] Setting key 4634f95343cbc1317184ffbeb2e56a58
I20260928 06:49:26.522800 28622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dc9ef1e37164a1abb28c63ea6fa614b: Bootstrap starting.
I20260928 06:49:26.525625 28622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dc9ef1e37164a1abb28c63ea6fa614b: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:26.526912 28622 log.cc:824] T 00000000000000000000000000000000 P 3dc9ef1e37164a1abb28c63ea6fa614b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:26.527927 28625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:26.523345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:56573 (local address 127.0.0.1:39727)
0928 06:49:26.524340 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:26.524360 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:26.524406 (+ 46us) server_negotiation.cc:407] Connection header received
0928 06:49:26.525051 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:26.525067 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:26.525435 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:26.525805 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:26.526630 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:26.526635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:26.526650 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:26.526668 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:26.526721 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:26.526740 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:26.526865 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:26.526972 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 06:49:26.527153 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":158,"threads_started":1}
I20260928 06:49:26.529203 28622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dc9ef1e37164a1abb28c63ea6fa614b: No bootstrap required, opened a new log
I20260928 06:49:26.530793 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:49:26.533543 28622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dc9ef1e37164a1abb28c63ea6fa614b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc9ef1e37164a1abb28c63ea6fa614b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39727 } }
I20260928 06:49:26.533809 28622 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dc9ef1e37164a1abb28c63ea6fa614b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:26.533876 28622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dc9ef1e37164a1abb28c63ea6fa614b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc9ef1e37164a1abb28c63ea6fa614b, State: Initialized, Role: FOLLOWER
I20260928 06:49:26.534385 28622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dc9ef1e37164a1abb28c63ea6fa614b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc9ef1e37164a1abb28c63ea6fa614b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39727 } }
I20260928 06:49:26.534535 28625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:26.532794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:60591 (local address 127.0.0.1:39727)
0928 06:49:26.532904 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:26.532909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:26.533112 (+ 203us) server_negotiation.cc:407] Connection header received
0928 06:49:26.533245 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:26.533248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:26.533281 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:26.533366 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:26.534140 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:26.534143 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:26.534146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:26.534157 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:26.534185 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:26.534203 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:26.534295 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:26.534369 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 06:49:26.534414 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:49:26.534577 28622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dc9ef1e37164a1abb28c63ea6fa614b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:49:26.534739 28622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dc9ef1e37164a1abb28c63ea6fa614b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:49:26.534870 28622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dc9ef1e37164a1abb28c63ea6fa614b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:26.536100 28622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dc9ef1e37164a1abb28c63ea6fa614b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc9ef1e37164a1abb28c63ea6fa614b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39727 } }
I20260928 06:49:26.536466 28622 leader_election.cc:302] T 00000000000000000000000000000000 P 3dc9ef1e37164a1abb28c63ea6fa614b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dc9ef1e37164a1abb28c63ea6fa614b; no voters:
I20260928 06:49:26.536794 28622 leader_election.cc:288] T 00000000000000000000000000000000 P 3dc9ef1e37164a1abb28c63ea6fa614b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:49:26.536931 28632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dc9ef1e37164a1abb28c63ea6fa614b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:26.537166 28632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dc9ef1e37164a1abb28c63ea6fa614b [term 1 LEADER]: Becoming Leader. State: Replica: 3dc9ef1e37164a1abb28c63ea6fa614b, State: Running, Role: LEADER
I20260928 06:49:26.537511 28632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dc9ef1e37164a1abb28c63ea6fa614b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc9ef1e37164a1abb28c63ea6fa614b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39727 } }
I20260928 06:49:26.538122 28622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dc9ef1e37164a1abb28c63ea6fa614b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:26.539186 28633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dc9ef1e37164a1abb28c63ea6fa614b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dc9ef1e37164a1abb28c63ea6fa614b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc9ef1e37164a1abb28c63ea6fa614b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39727 } } }
I20260928 06:49:26.539307 28633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dc9ef1e37164a1abb28c63ea6fa614b [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:26.539608 28634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dc9ef1e37164a1abb28c63ea6fa614b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dc9ef1e37164a1abb28c63ea6fa614b. Latest consensus state: current_term: 1 leader_uuid: "3dc9ef1e37164a1abb28c63ea6fa614b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc9ef1e37164a1abb28c63ea6fa614b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39727 } } }
I20260928 06:49:26.539726 28634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dc9ef1e37164a1abb28c63ea6fa614b [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:26.539747 28641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:26.543054 28641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:49:26.549433 28641 catalog_manager.cc:1347] Generated new cluster ID: 5a33c01f5d40469eafbd420952b39e19
I20260928 06:49:26.549507 28641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:26.570166 28641 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:26.570545 28641 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:26.578864 28641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dc9ef1e37164a1abb28c63ea6fa614b: Generated new TSK 0
I20260928 06:49:26.579717 28641 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:49:26.582901 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 28554
2026-09-28T06:49:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:49:26 dist-test-slave-twl6 krb5kdc[28653](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:49:26 dist-test-slave-twl6 krb5kdc[28653](info): set up 2 sockets
Sep 28 06:49:26 dist-test-slave-twl6 krb5kdc[28653](info): commencing operation
krb5kdc: starting...
W20260928 06:49:27.612665 7360 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:49:27 dist-test-slave-twl6 krb5kdc[28653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:49:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:49:27.735221 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43919
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:27.840009 28666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:27.840245 28666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:27.840296 28666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:27.843739 28666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:27.843782 28666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:27.843799 28666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:27.843817 28666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:27.843832 28666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:27.848608 28666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43919
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:27.849481 28666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:27.850409 28666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:27.858103 28671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:27.858114 28674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:27.858134 28672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:27.858207 28666 server_base.cc:1034] running on GCE node
I20260928 06:49:27.858850 28666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:27.859862 28666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:27.861049 28666 hybrid_clock.cc:648] HybridClock initialized: now 1790578167861038 us; error 22 us; skew 500 ppm
Sep 28 06:49:27 dist-test-slave-twl6 krb5kdc[28653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:27.867990 28666 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:49:27.868854 28666 webserver.cc:460] Webserver started at http://127.0.0.1:43331/ using document root <none> and password file <none>
I20260928 06:49:27.869402 28666 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:27.869499 28666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:27.869706 28666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:27.871659 28666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "c4fae729963f42f1b7e46669e1115928"
format_stamp: "Formatted at 2026-09-28 06:49:27 on dist-test-slave-twl6"
server_key: "c25812dcf9ca0d3696a5371efd727936"
server_key_iv: "760e751d6751dca7787833bc1f45b938"
server_key_version: "encryptionkey@0"
I20260928 06:49:27.872211 28666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "c4fae729963f42f1b7e46669e1115928"
format_stamp: "Formatted at 2026-09-28 06:49:27 on dist-test-slave-twl6"
server_key: "c25812dcf9ca0d3696a5371efd727936"
server_key_iv: "760e751d6751dca7787833bc1f45b938"
server_key_version: "encryptionkey@0"
I20260928 06:49:27.875762 28666 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:49:27.878044 28681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:27.879153 28666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:49:27.879283 28666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "c4fae729963f42f1b7e46669e1115928"
format_stamp: "Formatted at 2026-09-28 06:49:27 on dist-test-slave-twl6"
server_key: "c25812dcf9ca0d3696a5371efd727936"
server_key_iv: "760e751d6751dca7787833bc1f45b938"
server_key_version: "encryptionkey@0"
I20260928 06:49:27.879384 28666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:27.894141 28666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:27.897226 28666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:27.897426 28666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:27.903842 28666 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43919
I20260928 06:49:27.903877 28733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43919 every 8 connection(s)
I20260928 06:49:27.904814 28666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:27.907420 28734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:27.910653 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 28666
I20260928 06:49:27.910794 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:27.911108 7360 external_mini_cluster.cc:1428] Setting key e87238f6d3e0271cbc8f1d34d758531c
I20260928 06:49:27.912917 28734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4fae729963f42f1b7e46669e1115928: Bootstrap starting.
I20260928 06:49:27.915382 28734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4fae729963f42f1b7e46669e1115928: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:27.916209 28734 log.cc:824] T 00000000000000000000000000000000 P c4fae729963f42f1b7e46669e1115928: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:27.918145 28734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4fae729963f42f1b7e46669e1115928: No bootstrap required, opened a new log
Sep 28 06:49:27 dist-test-slave-twl6 krb5kdc[28653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:49:27.922075 28734 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4fae729963f42f1b7e46669e1115928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fae729963f42f1b7e46669e1115928" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43919 } }
I20260928 06:49:27.922343 28734 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4fae729963f42f1b7e46669e1115928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:27.922415 28734 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4fae729963f42f1b7e46669e1115928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4fae729963f42f1b7e46669e1115928, State: Initialized, Role: FOLLOWER
I20260928 06:49:27.922863 28734 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4fae729963f42f1b7e46669e1115928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fae729963f42f1b7e46669e1115928" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43919 } }
I20260928 06:49:27.922981 28734 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4fae729963f42f1b7e46669e1115928 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:49:27.923061 28734 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4fae729963f42f1b7e46669e1115928 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:49:27.923156 28734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4fae729963f42f1b7e46669e1115928 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:27.924226 28734 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4fae729963f42f1b7e46669e1115928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fae729963f42f1b7e46669e1115928" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43919 } }
I20260928 06:49:27.924614 28734 leader_election.cc:302] T 00000000000000000000000000000000 P c4fae729963f42f1b7e46669e1115928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4fae729963f42f1b7e46669e1115928; no voters:
I20260928 06:49:27.924963 28734 leader_election.cc:288] T 00000000000000000000000000000000 P c4fae729963f42f1b7e46669e1115928 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:49:27.925057 28739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4fae729963f42f1b7e46669e1115928 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:27.925314 28739 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4fae729963f42f1b7e46669e1115928 [term 1 LEADER]: Becoming Leader. State: Replica: c4fae729963f42f1b7e46669e1115928, State: Running, Role: LEADER
I20260928 06:49:27.925635 28739 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4fae729963f42f1b7e46669e1115928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fae729963f42f1b7e46669e1115928" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43919 } }
I20260928 06:49:27.925827 28737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:27.912368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:42421 (local address 127.0.0.1:43919)
0928 06:49:27.912740 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:27.912760 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:27.912793 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:49:27.913417 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:27.913445 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:27.913777 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:27.914057 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:27.914921 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:27.915978 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:27.916959 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:27.917370 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:27.917477 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:27.920123 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:27.920129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:27.920145 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:27.920174 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:27.922580 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:27.923129 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:27.923134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:27.923139 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:27.923292 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:27.923600 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:27.923603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:27.923605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:27.924158 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:27.924376 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:27.924682 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 06:49:27.924908 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":126,"threads_started":1}
I20260928 06:49:27.926173 28734 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4fae729963f42f1b7e46669e1115928 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:27.927259 28741 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4fae729963f42f1b7e46669e1115928 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4fae729963f42f1b7e46669e1115928. Latest consensus state: current_term: 1 leader_uuid: "c4fae729963f42f1b7e46669e1115928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fae729963f42f1b7e46669e1115928" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43919 } } }
I20260928 06:49:27.927377 28741 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4fae729963f42f1b7e46669e1115928 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:27.927701 28740 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4fae729963f42f1b7e46669e1115928 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4fae729963f42f1b7e46669e1115928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fae729963f42f1b7e46669e1115928" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43919 } } }
I20260928 06:49:27.927762 28748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:27.927778 28740 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4fae729963f42f1b7e46669e1115928 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:27.929068 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:49:27.930542 28748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:49:27.935218 28748 catalog_manager.cc:1347] Generated new cluster ID: 7aa8bd53b4a144c0a5e9656e0612ed76
I20260928 06:49:27.935272 28748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:27.944545 28737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:27.935490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:36613 (local address 127.0.0.1:43919)
0928 06:49:27.935584 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:27.935589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:27.935868 (+ 279us) server_negotiation.cc:407] Connection header received
0928 06:49:27.936036 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:27.936039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:27.936090 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:27.936170 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:27.937026 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:27.937781 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:27.938860 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:27.939208 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:27.939289 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:27.941091 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:27.941094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:27.941097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:27.941119 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:27.942726 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:27.943241 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:27.943244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:27.943246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:27.943315 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:27.943633 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:27.943636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:27.943637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:27.943831 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:27.944002 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:27.944300 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 06:49:27.944392 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:49:27.947465 28748 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:27.948385 28748 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:27.963129 28748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4fae729963f42f1b7e46669e1115928: Generated new TSK 0
I20260928 06:49:27.963821 28748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:49:27.971207 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 28666
W20260928 06:49:27.976619 28651 connection.cc:535] client connection to 127.0.0.1:43919 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43919): Connection reset by peer (error 104)
2026-09-28T06:49:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:49:28 dist-test-slave-twl6 krb5kdc[28765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:49:28 dist-test-slave-twl6 krb5kdc[28765](info): set up 2 sockets
Sep 28 06:49:28 dist-test-slave-twl6 krb5kdc[28765](info): commencing operation
krb5kdc: starting...
W20260928 06:49:28.980393 7360 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:49:29 dist-test-slave-twl6 krb5kdc[28765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:49:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:49:29.098093 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35283
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:29.203780 28778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:29.204133 28778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:29.204219 28778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:29.208077 28778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:29.208160 28778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:29.208197 28778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:29.208232 28778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:29.208269 28778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:29.213366 28778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35283
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:29.214437 28778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:29.215453 28778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:29.223624 28786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:29.223641 28783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:29.223621 28784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:29.223807 28778 server_base.cc:1034] running on GCE node
I20260928 06:49:29.224524 28778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:29.225564 28778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:29.226755 28778 hybrid_clock.cc:648] HybridClock initialized: now 1790578169226731 us; error 40 us; skew 500 ppm
Sep 28 06:49:29 dist-test-slave-twl6 krb5kdc[28765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:29.229391 28778 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:49:29.230161 28778 webserver.cc:460] Webserver started at http://127.0.0.1:46425/ using document root <none> and password file <none>
I20260928 06:49:29.230676 28778 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:29.230787 28778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:29.231011 28778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:29.233004 28778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "e345733fbd1c41eaafea6300a31bbf7c"
format_stamp: "Formatted at 2026-09-28 06:49:29 on dist-test-slave-twl6"
server_key: "5cb783d7ac73049748f1ce0670a95fac"
server_key_iv: "d9e0493c05b9ead8749ede3950f3ba77"
server_key_version: "encryptionkey@0"
I20260928 06:49:29.233577 28778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "e345733fbd1c41eaafea6300a31bbf7c"
format_stamp: "Formatted at 2026-09-28 06:49:29 on dist-test-slave-twl6"
server_key: "5cb783d7ac73049748f1ce0670a95fac"
server_key_iv: "d9e0493c05b9ead8749ede3950f3ba77"
server_key_version: "encryptionkey@0"
I20260928 06:49:29.237115 28778 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:49:29.239285 28793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:29.240442 28778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:49:29.240574 28778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "e345733fbd1c41eaafea6300a31bbf7c"
format_stamp: "Formatted at 2026-09-28 06:49:29 on dist-test-slave-twl6"
server_key: "5cb783d7ac73049748f1ce0670a95fac"
server_key_iv: "d9e0493c05b9ead8749ede3950f3ba77"
server_key_version: "encryptionkey@0"
I20260928 06:49:29.240686 28778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:29.260334 28778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:29.263317 28778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:29.263525 28778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:29.270495 28778 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35283
I20260928 06:49:29.270565 28845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35283 every 8 connection(s)
I20260928 06:49:29.271441 28778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:29.273707 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 28778
I20260928 06:49:29.273855 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:29.274170 7360 external_mini_cluster.cc:1428] Setting key 769da9fd86592ebd62dbe42c5a837586
I20260928 06:49:29.274302 28846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:29.281342 28846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e345733fbd1c41eaafea6300a31bbf7c: Bootstrap starting.
I20260928 06:49:29.284137 28846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e345733fbd1c41eaafea6300a31bbf7c: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:29.284294 28849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:29.275518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:33571 (local address 127.0.0.1:35283)
0928 06:49:29.275953 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:29.275971 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:29.276136 (+ 165us) server_negotiation.cc:407] Connection header received
0928 06:49:29.277000 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:29.277024 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:29.277448 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:29.277834 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:29.278218 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:29.279628 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:29.280550 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:29.281188 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:29.281349 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:29.282288 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:29.282294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:29.282308 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:29.282836 (+ 528us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:29.282909 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:29.282941 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:29.283068 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:29.283204 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 06:49:29.283330 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":112,"threads_started":1}
I20260928 06:49:29.285020 28846 log.cc:824] T 00000000000000000000000000000000 P e345733fbd1c41eaafea6300a31bbf7c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:29.287194 28846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e345733fbd1c41eaafea6300a31bbf7c: No bootstrap required, opened a new log
I20260928 06:49:29.287261 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:49:29.291404 28846 raft_consensus.cc:348] T 00000000000000000000000000000000 P e345733fbd1c41eaafea6300a31bbf7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e345733fbd1c41eaafea6300a31bbf7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35283 } }
I20260928 06:49:29.291694 28846 raft_consensus.cc:374] T 00000000000000000000000000000000 P e345733fbd1c41eaafea6300a31bbf7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:29.291770 28846 raft_consensus.cc:729] T 00000000000000000000000000000000 P e345733fbd1c41eaafea6300a31bbf7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e345733fbd1c41eaafea6300a31bbf7c, State: Initialized, Role: FOLLOWER
I20260928 06:49:29.292302 28846 consensus_queue.cc:260] T 00000000000000000000000000000000 P e345733fbd1c41eaafea6300a31bbf7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e345733fbd1c41eaafea6300a31bbf7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35283 } }
I20260928 06:49:29.292479 28846 raft_consensus.cc:388] T 00000000000000000000000000000000 P e345733fbd1c41eaafea6300a31bbf7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:49:29.292559 28846 raft_consensus.cc:482] T 00000000000000000000000000000000 P e345733fbd1c41eaafea6300a31bbf7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:49:29.292667 28846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e345733fbd1c41eaafea6300a31bbf7c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:29.293623 28849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:29.289409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:50667 (local address 127.0.0.1:35283)
0928 06:49:29.289526 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:29.289530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:29.289690 (+ 160us) server_negotiation.cc:407] Connection header received
0928 06:49:29.289804 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:29.289807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:29.289844 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:29.289909 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:29.290197 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:29.290730 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:29.291821 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:29.292131 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:29.292213 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:29.293126 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:29.293128 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:29.293130 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:29.293141 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:29.293165 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:29.293175 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:29.293244 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:29.293407 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 06:49:29.293465 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:49:29.293917 28846 raft_consensus.cc:504] T 00000000000000000000000000000000 P e345733fbd1c41eaafea6300a31bbf7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e345733fbd1c41eaafea6300a31bbf7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35283 } }
I20260928 06:49:29.294279 28846 leader_election.cc:302] T 00000000000000000000000000000000 P e345733fbd1c41eaafea6300a31bbf7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e345733fbd1c41eaafea6300a31bbf7c; no voters:
I20260928 06:49:29.294631 28846 leader_election.cc:288] T 00000000000000000000000000000000 P e345733fbd1c41eaafea6300a31bbf7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:49:29.294727 28856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e345733fbd1c41eaafea6300a31bbf7c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:29.294951 28856 raft_consensus.cc:686] T 00000000000000000000000000000000 P e345733fbd1c41eaafea6300a31bbf7c [term 1 LEADER]: Becoming Leader. State: Replica: e345733fbd1c41eaafea6300a31bbf7c, State: Running, Role: LEADER
I20260928 06:49:29.295281 28856 consensus_queue.cc:237] T 00000000000000000000000000000000 P e345733fbd1c41eaafea6300a31bbf7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e345733fbd1c41eaafea6300a31bbf7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35283 } }
I20260928 06:49:29.296036 28846 sys_catalog.cc:564] T 00000000000000000000000000000000 P e345733fbd1c41eaafea6300a31bbf7c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:29.297043 28857 sys_catalog.cc:455] T 00000000000000000000000000000000 P e345733fbd1c41eaafea6300a31bbf7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e345733fbd1c41eaafea6300a31bbf7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e345733fbd1c41eaafea6300a31bbf7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35283 } } }
I20260928 06:49:29.297192 28857 sys_catalog.cc:458] T 00000000000000000000000000000000 P e345733fbd1c41eaafea6300a31bbf7c [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:29.297624 28858 sys_catalog.cc:455] T 00000000000000000000000000000000 P e345733fbd1c41eaafea6300a31bbf7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e345733fbd1c41eaafea6300a31bbf7c. Latest consensus state: current_term: 1 leader_uuid: "e345733fbd1c41eaafea6300a31bbf7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e345733fbd1c41eaafea6300a31bbf7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35283 } } }
I20260928 06:49:29.297713 28858 sys_catalog.cc:458] T 00000000000000000000000000000000 P e345733fbd1c41eaafea6300a31bbf7c [sys.catalog]: This master's current role is: LEADER
W20260928 06:49:29.300459 28871 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e345733fbd1c41eaafea6300a31bbf7c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:49:29.300552 28871 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:49:29.300624 28865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:29.301234 28865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:49:29.306715 28865 catalog_manager.cc:1347] Generated new cluster ID: 68ffd11bd5cb4032915100b00830739d
I20260928 06:49:29.306782 28865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:29.324601 28865 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:29.325830 28865 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:29.333248 28865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e345733fbd1c41eaafea6300a31bbf7c: Generated new TSK 0
I20260928 06:49:29.334089 28865 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:49:29.346966 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 28778
W20260928 06:49:29.352414 28763 connection.cc:535] client connection to 127.0.0.1:35283 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35283): Connection reset by peer (error 104)
2026-09-28T06:49:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1375 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:49:29 dist-test-slave-twl6 krb5kdc[28877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:49:29 dist-test-slave-twl6 krb5kdc[28877](info): set up 2 sockets
Sep 28 06:49:29 dist-test-slave-twl6 krb5kdc[28877](info): commencing operation
krb5kdc: starting...
W20260928 06:49:30.344420 7360 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:49:30 dist-test-slave-twl6 krb5kdc[28877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:49:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:49:30.460495 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35735
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:30.570580 28890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:30.570840 28890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:30.570919 28890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:30.574988 28890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:30.575052 28890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:30.575073 28890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:30.575088 28890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:30.575103 28890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:30.580353 28890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35735
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:30.581379 28890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:30.582403 28890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:30.590756 28896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:30.590798 28890 server_base.cc:1034] running on GCE node
W20260928 06:49:30.590763 28895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:30.590771 28898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:30.591534 28890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:30.592617 28890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:30.593775 28890 hybrid_clock.cc:648] HybridClock initialized: now 1790578170593748 us; error 39 us; skew 500 ppm
Sep 28 06:49:30 dist-test-slave-twl6 krb5kdc[28877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:30.596411 28890 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:49:30.597191 28890 webserver.cc:460] Webserver started at http://127.0.0.1:41847/ using document root <none> and password file <none>
I20260928 06:49:30.597733 28890 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:30.597806 28890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:30.597962 28890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:30.600055 28890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "c4fccec4ff054fab94f6a02657191d68"
format_stamp: "Formatted at 2026-09-28 06:49:30 on dist-test-slave-twl6"
server_key: "279056907c9a3aa9ed9f2a2573bc8036"
server_key_iv: "db33fc16881d790648d16c1ce47507ff"
server_key_version: "encryptionkey@0"
I20260928 06:49:30.600538 28890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "c4fccec4ff054fab94f6a02657191d68"
format_stamp: "Formatted at 2026-09-28 06:49:30 on dist-test-slave-twl6"
server_key: "279056907c9a3aa9ed9f2a2573bc8036"
server_key_iv: "db33fc16881d790648d16c1ce47507ff"
server_key_version: "encryptionkey@0"
I20260928 06:49:30.604002 28890 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:49:30.606168 28905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:30.607311 28890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:49:30.607414 28890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "c4fccec4ff054fab94f6a02657191d68"
format_stamp: "Formatted at 2026-09-28 06:49:30 on dist-test-slave-twl6"
server_key: "279056907c9a3aa9ed9f2a2573bc8036"
server_key_iv: "db33fc16881d790648d16c1ce47507ff"
server_key_version: "encryptionkey@0"
I20260928 06:49:30.607491 28890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:30.611713 28890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:30.614399 28890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:30.614539 28890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:30.621037 28890 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35735
I20260928 06:49:30.621099 28957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35735 every 8 connection(s)
I20260928 06:49:30.621904 28890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:30.624493 28958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:30.626170 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 28890
I20260928 06:49:30.626328 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:30.626598 7360 external_mini_cluster.cc:1428] Setting key 0dba7cba56b01083c7b5000f5996aa1c
I20260928 06:49:30.630903 28958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4fccec4ff054fab94f6a02657191d68: Bootstrap starting.
I20260928 06:49:30.631976 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:30.627781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:34459 (local address 127.0.0.1:35735)
0928 06:49:30.628204 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:30.628239 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:30.628280 (+ 41us) server_negotiation.cc:407] Connection header received
0928 06:49:30.629069 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:30.629093 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:30.629464 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:30.629864 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:30.630634 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:30.630639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:30.630654 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:30.630665 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:30.630725 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:30.630751 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:30.630885 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:30.630951 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 06:49:30.631123 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":150,"threads_started":1}
I20260928 06:49:30.633490 28958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4fccec4ff054fab94f6a02657191d68: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:30.634310 28958 log.cc:824] T 00000000000000000000000000000000 P c4fccec4ff054fab94f6a02657191d68: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:30.634754 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:49:30.636234 28958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4fccec4ff054fab94f6a02657191d68: No bootstrap required, opened a new log
I20260928 06:49:30.638291 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:30.636741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:42407 (local address 127.0.0.1:35735)
0928 06:49:30.636846 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:30.636851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:30.637044 (+ 193us) server_negotiation.cc:407] Connection header received
0928 06:49:30.637175 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:30.637178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:30.637212 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:30.637290 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:30.637934 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:30.637937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:30.637940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:30.637950 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:30.637974 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:30.637987 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:30.638079 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:30.638150 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 06:49:30.638188 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:49:30.639652 28958 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4fccec4ff054fab94f6a02657191d68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fccec4ff054fab94f6a02657191d68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35735 } }
I20260928 06:49:30.639845 28958 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4fccec4ff054fab94f6a02657191d68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:30.639938 28958 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4fccec4ff054fab94f6a02657191d68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4fccec4ff054fab94f6a02657191d68, State: Initialized, Role: FOLLOWER
I20260928 06:49:30.640318 28958 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4fccec4ff054fab94f6a02657191d68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fccec4ff054fab94f6a02657191d68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35735 } }
I20260928 06:49:30.640465 28958 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4fccec4ff054fab94f6a02657191d68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:49:30.640527 28958 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4fccec4ff054fab94f6a02657191d68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:49:30.640625 28958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4fccec4ff054fab94f6a02657191d68 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:30.641417 28958 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4fccec4ff054fab94f6a02657191d68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fccec4ff054fab94f6a02657191d68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35735 } }
I20260928 06:49:30.641732 28958 leader_election.cc:302] T 00000000000000000000000000000000 P c4fccec4ff054fab94f6a02657191d68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4fccec4ff054fab94f6a02657191d68; no voters:
I20260928 06:49:30.642002 28958 leader_election.cc:288] T 00000000000000000000000000000000 P c4fccec4ff054fab94f6a02657191d68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:49:30.642122 28968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4fccec4ff054fab94f6a02657191d68 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:30.642338 28968 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4fccec4ff054fab94f6a02657191d68 [term 1 LEADER]: Becoming Leader. State: Replica: c4fccec4ff054fab94f6a02657191d68, State: Running, Role: LEADER
I20260928 06:49:30.642680 28968 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4fccec4ff054fab94f6a02657191d68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fccec4ff054fab94f6a02657191d68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35735 } }
I20260928 06:49:30.643136 28958 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4fccec4ff054fab94f6a02657191d68 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:30.644479 28969 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4fccec4ff054fab94f6a02657191d68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4fccec4ff054fab94f6a02657191d68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fccec4ff054fab94f6a02657191d68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35735 } } }
I20260928 06:49:30.644510 28970 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4fccec4ff054fab94f6a02657191d68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4fccec4ff054fab94f6a02657191d68. Latest consensus state: current_term: 1 leader_uuid: "c4fccec4ff054fab94f6a02657191d68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fccec4ff054fab94f6a02657191d68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35735 } } }
I20260928 06:49:30.644606 28970 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4fccec4ff054fab94f6a02657191d68 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:30.644606 28969 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4fccec4ff054fab94f6a02657191d68 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:30.645601 28978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:30.648347 28978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:49:30.653402 28978 catalog_manager.cc:1347] Generated new cluster ID: 0082b00c40694a13a2f3f5eddd03d793
I20260928 06:49:30.653484 28978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:30.672461 28978 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:30.672753 28978 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:30.678902 28978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4fccec4ff054fab94f6a02657191d68: Generated new TSK 0
I20260928 06:49:30.679649 28978 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:49:30.683118 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 28890
2026-09-28T06:49:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:49:30 dist-test-slave-twl6 krb5kdc[28989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:49:30 dist-test-slave-twl6 krb5kdc[28989](info): set up 2 sockets
Sep 28 06:49:30 dist-test-slave-twl6 krb5kdc[28989](info): commencing operation
krb5kdc: starting...
W20260928 06:49:31.684100 7360 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:49:31 dist-test-slave-twl6 krb5kdc[28989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:49:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:49:31.794924 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:42985
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:42985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:31.897341 29002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:31.897558 29002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:31.897614 29002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:31.901507 29002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:31.901553 29002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:31.901571 29002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:31.901613 29002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:31.901683 29002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:31.906457 29002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:42985
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:42985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:31.907397 29002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:31.908392 29002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:31.916220 29007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:31.916226 29008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:31.916298 29002 server_base.cc:1034] running on GCE node
W20260928 06:49:31.916226 29010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:31.916915 29002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:31.917869 29002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:31.919026 29002 hybrid_clock.cc:648] HybridClock initialized: now 1790578171918999 us; error 38 us; skew 500 ppm
Sep 28 06:49:31 dist-test-slave-twl6 krb5kdc[28989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578171, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:31.921468 29002 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:49:31.922230 29002 webserver.cc:460] Webserver started at http://127.7.48.62:43919/ using document root <none> and password file <none>
I20260928 06:49:31.922765 29002 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:31.922822 29002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:31.923048 29002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:31.925045 29002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "e12a6939ec0d40bb81f9dd77a7428225"
format_stamp: "Formatted at 2026-09-28 06:49:31 on dist-test-slave-twl6"
server_key: "4328a2f417297fb9ff667da5b082f697"
server_key_iv: "afa570d301eb8d66363e10a514c90c5f"
server_key_version: "encryptionkey@0"
I20260928 06:49:31.925587 29002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "e12a6939ec0d40bb81f9dd77a7428225"
format_stamp: "Formatted at 2026-09-28 06:49:31 on dist-test-slave-twl6"
server_key: "4328a2f417297fb9ff667da5b082f697"
server_key_iv: "afa570d301eb8d66363e10a514c90c5f"
server_key_version: "encryptionkey@0"
I20260928 06:49:31.929046 29002 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:49:31.931087 29017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:31.932287 29002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:49:31.932411 29002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "e12a6939ec0d40bb81f9dd77a7428225"
format_stamp: "Formatted at 2026-09-28 06:49:31 on dist-test-slave-twl6"
server_key: "4328a2f417297fb9ff667da5b082f697"
server_key_iv: "afa570d301eb8d66363e10a514c90c5f"
server_key_version: "encryptionkey@0"
I20260928 06:49:31.932507 29002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:31.965891 29002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:31.968761 29002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:31.968986 29002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:31.975661 29002 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:42985
I20260928 06:49:31.975719 29069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:42985 every 8 connection(s)
I20260928 06:49:31.976564 29002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:31.979064 29070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:31.980495 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 29002
I20260928 06:49:31.980633 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:31.980912 7360 external_mini_cluster.cc:1428] Setting key 690288de3d035593d54c578f9aa8dcbd
I20260928 06:49:31.985131 29070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e12a6939ec0d40bb81f9dd77a7428225: Bootstrap starting.
I20260928 06:49:31.987473 29070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e12a6939ec0d40bb81f9dd77a7428225: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:31.988184 29070 log.cc:824] T 00000000000000000000000000000000 P e12a6939ec0d40bb81f9dd77a7428225: Log is configured to *not* fsync() on all Append() calls
Sep 28 06:49:31 dist-test-slave-twl6 krb5kdc[28989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:49:31.990057 29070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e12a6939ec0d40bb81f9dd77a7428225: No bootstrap required, opened a new log
I20260928 06:49:31.994484 29073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:31.982142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37219 (local address 127.7.48.62:42985)
0928 06:49:31.982501 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:31.982516 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:31.982547 (+ 31us) server_negotiation.cc:407] Connection header received
0928 06:49:31.983115 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:31.983130 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:31.983449 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:31.983703 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:31.984617 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:31.985604 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:31.986638 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:31.987038 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:31.987144 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:31.989681 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:31.989687 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:31.989700 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:31.989727 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:31.991488 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:31.991950 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:31.991958 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:31.991964 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:31.992103 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:31.992453 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:31.992456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:31.992459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:31.992916 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:31.993182 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:31.993347 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 06:49:31.993571 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":116,"threads_started":1}
I20260928 06:49:31.994514 29070 raft_consensus.cc:348] T 00000000000000000000000000000000 P e12a6939ec0d40bb81f9dd77a7428225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12a6939ec0d40bb81f9dd77a7428225" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42985 } }
I20260928 06:49:31.994760 29070 raft_consensus.cc:374] T 00000000000000000000000000000000 P e12a6939ec0d40bb81f9dd77a7428225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:31.994817 29070 raft_consensus.cc:729] T 00000000000000000000000000000000 P e12a6939ec0d40bb81f9dd77a7428225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e12a6939ec0d40bb81f9dd77a7428225, State: Initialized, Role: FOLLOWER
I20260928 06:49:31.995249 29070 consensus_queue.cc:260] T 00000000000000000000000000000000 P e12a6939ec0d40bb81f9dd77a7428225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12a6939ec0d40bb81f9dd77a7428225" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42985 } }
I20260928 06:49:31.995404 29070 raft_consensus.cc:388] T 00000000000000000000000000000000 P e12a6939ec0d40bb81f9dd77a7428225 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:49:31.995463 29070 raft_consensus.cc:482] T 00000000000000000000000000000000 P e12a6939ec0d40bb81f9dd77a7428225 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:49:31.995550 29070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e12a6939ec0d40bb81f9dd77a7428225 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:31.996421 29070 raft_consensus.cc:504] T 00000000000000000000000000000000 P e12a6939ec0d40bb81f9dd77a7428225 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12a6939ec0d40bb81f9dd77a7428225" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42985 } }
I20260928 06:49:31.996747 29070 leader_election.cc:302] T 00000000000000000000000000000000 P e12a6939ec0d40bb81f9dd77a7428225 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e12a6939ec0d40bb81f9dd77a7428225; no voters:
I20260928 06:49:31.997103 29070 leader_election.cc:288] T 00000000000000000000000000000000 P e12a6939ec0d40bb81f9dd77a7428225 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:49:31.997182 29075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e12a6939ec0d40bb81f9dd77a7428225 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:31.997227 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:49:31.997382 29075 raft_consensus.cc:686] T 00000000000000000000000000000000 P e12a6939ec0d40bb81f9dd77a7428225 [term 1 LEADER]: Becoming Leader. State: Replica: e12a6939ec0d40bb81f9dd77a7428225, State: Running, Role: LEADER
I20260928 06:49:31.997627 29075 consensus_queue.cc:237] T 00000000000000000000000000000000 P e12a6939ec0d40bb81f9dd77a7428225 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12a6939ec0d40bb81f9dd77a7428225" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42985 } }
I20260928 06:49:31.998065 29070 sys_catalog.cc:564] T 00000000000000000000000000000000 P e12a6939ec0d40bb81f9dd77a7428225 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:31.998710 29075 sys_catalog.cc:455] T 00000000000000000000000000000000 P e12a6939ec0d40bb81f9dd77a7428225 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e12a6939ec0d40bb81f9dd77a7428225. Latest consensus state: current_term: 1 leader_uuid: "e12a6939ec0d40bb81f9dd77a7428225" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12a6939ec0d40bb81f9dd77a7428225" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42985 } } }
I20260928 06:49:31.998806 29075 sys_catalog.cc:458] T 00000000000000000000000000000000 P e12a6939ec0d40bb81f9dd77a7428225 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:31.999085 29076 sys_catalog.cc:455] T 00000000000000000000000000000000 P e12a6939ec0d40bb81f9dd77a7428225 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e12a6939ec0d40bb81f9dd77a7428225" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12a6939ec0d40bb81f9dd77a7428225" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 42985 } } }
I20260928 06:49:31.999169 29076 sys_catalog.cc:458] T 00000000000000000000000000000000 P e12a6939ec0d40bb81f9dd77a7428225 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:31.999939 29087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:32.002938 29087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:49:32.007553 29073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:31.999443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55019 (local address 127.7.48.62:42985)
0928 06:49:31.999547 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:31.999551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:31.999637 (+ 86us) server_negotiation.cc:407] Connection header received
0928 06:49:31.999824 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:31.999829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:31.999869 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:31.999946 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:32.000786 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:32.001387 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:32.002395 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:32.002710 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:32.002816 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:32.004296 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:32.004300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:32.004302 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:32.004327 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:32.005813 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:32.006268 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:32.006271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:32.006273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:32.006343 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:32.006650 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:32.006653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:32.006654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:32.006839 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:32.006971 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:32.007317 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 06:49:32.007407 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:49:32.008970 29087 catalog_manager.cc:1347] Generated new cluster ID: dc9915efe0e24d8eb0f31dff4395dc37
I20260928 06:49:32.009024 29087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:32.023859 29087 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:32.024699 29087 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:32.030987 29087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e12a6939ec0d40bb81f9dd77a7428225: Generated new TSK 0
I20260928 06:49:32.031553 29087 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:49:32.038533 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 29002
W20260928 06:49:32.043803 28987 connection.cc:535] client connection to 127.7.48.62:42985 recv error: Network error: failed to read from TLS socket (remote: 127.7.48.62:42985): Connection reset by peer (error 104)
2026-09-28T06:49:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1355 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:49:32 dist-test-slave-twl6 krb5kdc[29101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:49:32 dist-test-slave-twl6 krb5kdc[29101](info): set up 2 sockets
Sep 28 06:49:32 dist-test-slave-twl6 krb5kdc[29101](info): commencing operation
krb5kdc: starting...
W20260928 06:49:33.044466 7360 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:49:33 dist-test-slave-twl6 krb5kdc[29101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:49:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:49:33.161048 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:44129
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:41287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:44129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:33.269040 29114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:33.269299 29114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:33.269378 29114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:33.273392 29114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:33.273473 29114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:33.273494 29114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:33.273511 29114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:33.273526 29114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:33.278795 29114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:41287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:44129
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:44129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:33.279850 29114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:33.280946 29114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:33.289291 29120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:33.289255 29119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:33.289312 29122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:33.289602 29114 server_base.cc:1034] running on GCE node
I20260928 06:49:33.290197 29114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:33.291261 29114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:33.292425 29114 hybrid_clock.cc:648] HybridClock initialized: now 1790578173292406 us; error 32 us; skew 500 ppm
Sep 28 06:49:33 dist-test-slave-twl6 krb5kdc[29101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578173, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:33.298375 29114 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:49:33.299309 29114 webserver.cc:460] Webserver started at http://127.7.48.62:46389/ using document root <none> and password file <none>
I20260928 06:49:33.299942 29114 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:33.300017 29114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:33.300271 29114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:33.302318 29114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "02cc6831c107494d92efce7d1da0053a"
format_stamp: "Formatted at 2026-09-28 06:49:33 on dist-test-slave-twl6"
server_key: "adeb93b16e3b2f83bcc96bbfddb65843"
server_key_iv: "cc0259e6c42549a200c554d59b7162f6"
server_key_version: "encryptionkey@0"
I20260928 06:49:33.302883 29114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "02cc6831c107494d92efce7d1da0053a"
format_stamp: "Formatted at 2026-09-28 06:49:33 on dist-test-slave-twl6"
server_key: "adeb93b16e3b2f83bcc96bbfddb65843"
server_key_iv: "cc0259e6c42549a200c554d59b7162f6"
server_key_version: "encryptionkey@0"
I20260928 06:49:33.307282 29114 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 06:49:33.309775 29129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:33.311019 29114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:49:33.311167 29114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "02cc6831c107494d92efce7d1da0053a"
format_stamp: "Formatted at 2026-09-28 06:49:33 on dist-test-slave-twl6"
server_key: "adeb93b16e3b2f83bcc96bbfddb65843"
server_key_iv: "cc0259e6c42549a200c554d59b7162f6"
server_key_version: "encryptionkey@0"
I20260928 06:49:33.311285 29114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:33.325290 29114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:33.328140 29114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:33.328338 29114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:33.335397 29114 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:44129
I20260928 06:49:33.335428 29181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:44129 every 8 connection(s)
I20260928 06:49:33.336458 29114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:33.336760 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 29114
I20260928 06:49:33.336912 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:33.337252 7360 external_mini_cluster.cc:1428] Setting key 87c1b99b441105a996e34195f79c7269
I20260928 06:49:33.339852 29182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:33.346637 29182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02cc6831c107494d92efce7d1da0053a: Bootstrap starting.
I20260928 06:49:33.347422 29185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:33.338633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42585 (local address 127.7.48.62:44129)
0928 06:49:33.339041 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:33.339060 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:33.339102 (+ 42us) server_negotiation.cc:407] Connection header received
0928 06:49:33.340027 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:33.340051 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:33.340472 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:33.340876 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:33.341249 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:33.342698 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:33.343721 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:33.344415 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:33.344592 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:33.345465 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:33.345471 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:33.345486 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:33.346006 (+ 520us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:33.346070 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:33.346099 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:33.346225 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:33.346425 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 06:49:33.346549 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":126,"threads_started":1}
I20260928 06:49:33.349141 29182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02cc6831c107494d92efce7d1da0053a: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:33.349998 29182 log.cc:824] T 00000000000000000000000000000000 P 02cc6831c107494d92efce7d1da0053a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:33.350522 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:49:33.352383 29182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02cc6831c107494d92efce7d1da0053a: No bootstrap required, opened a new log
I20260928 06:49:33.357012 29182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02cc6831c107494d92efce7d1da0053a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02cc6831c107494d92efce7d1da0053a" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 44129 } }
I20260928 06:49:33.357246 29185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:33.352829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52379 (local address 127.7.48.62:44129)
0928 06:49:33.352964 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:33.352969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:33.353168 (+ 199us) server_negotiation.cc:407] Connection header received
0928 06:49:33.353279 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:33.353281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:33.353319 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:33.353390 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:33.353705 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:33.354265 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:33.355469 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:33.355699 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:33.355783 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:33.356715 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:33.356718 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:33.356720 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:33.356732 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:33.356757 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:33.356768 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:33.356843 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:33.357006 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 06:49:33.357064 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 06:49:33.357273 29182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02cc6831c107494d92efce7d1da0053a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:33.357379 29182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02cc6831c107494d92efce7d1da0053a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02cc6831c107494d92efce7d1da0053a, State: Initialized, Role: FOLLOWER
I20260928 06:49:33.357816 29182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02cc6831c107494d92efce7d1da0053a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02cc6831c107494d92efce7d1da0053a" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 44129 } }
I20260928 06:49:33.357935 29182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02cc6831c107494d92efce7d1da0053a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:49:33.358007 29182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02cc6831c107494d92efce7d1da0053a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:49:33.358160 29182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02cc6831c107494d92efce7d1da0053a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:33.359099 29182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02cc6831c107494d92efce7d1da0053a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02cc6831c107494d92efce7d1da0053a" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 44129 } }
I20260928 06:49:33.359452 29182 leader_election.cc:302] T 00000000000000000000000000000000 P 02cc6831c107494d92efce7d1da0053a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02cc6831c107494d92efce7d1da0053a; no voters:
I20260928 06:49:33.359738 29182 leader_election.cc:288] T 00000000000000000000000000000000 P 02cc6831c107494d92efce7d1da0053a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:49:33.359869 29192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02cc6831c107494d92efce7d1da0053a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:33.360229 29192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02cc6831c107494d92efce7d1da0053a [term 1 LEADER]: Becoming Leader. State: Replica: 02cc6831c107494d92efce7d1da0053a, State: Running, Role: LEADER
I20260928 06:49:33.360549 29192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02cc6831c107494d92efce7d1da0053a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02cc6831c107494d92efce7d1da0053a" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 44129 } }
I20260928 06:49:33.361032 29182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02cc6831c107494d92efce7d1da0053a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:33.362324 29194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02cc6831c107494d92efce7d1da0053a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02cc6831c107494d92efce7d1da0053a. Latest consensus state: current_term: 1 leader_uuid: "02cc6831c107494d92efce7d1da0053a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02cc6831c107494d92efce7d1da0053a" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 44129 } } }
I20260928 06:49:33.362448 29194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02cc6831c107494d92efce7d1da0053a [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:33.362892 29202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:33.362866 29193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02cc6831c107494d92efce7d1da0053a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02cc6831c107494d92efce7d1da0053a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02cc6831c107494d92efce7d1da0053a" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 44129 } } }
I20260928 06:49:33.362959 29193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02cc6831c107494d92efce7d1da0053a [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:33.366474 29202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:49:33.371955 29202 catalog_manager.cc:1347] Generated new cluster ID: dfd4e12c6c9b47f48d0c7f764eeecdf7
I20260928 06:49:33.372021 29202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:33.383319 29202 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:33.384436 29202 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:33.392752 29202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02cc6831c107494d92efce7d1da0053a: Generated new TSK 0
I20260928 06:49:33.393460 29202 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:49:33.401609 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 29114
W20260928 06:49:33.407541 29099 connection.cc:535] client connection to 127.7.48.62:44129 recv error: Network error: failed to read from TLS socket (remote: 127.7.48.62:44129): Connection reset by peer (error 104)
2026-09-28T06:49:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1364 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:49:33 dist-test-slave-twl6 krb5kdc[29213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:49:33 dist-test-slave-twl6 krb5kdc[29213](info): set up 2 sockets
Sep 28 06:49:33 dist-test-slave-twl6 krb5kdc[29213](info): commencing operation
krb5kdc: starting...
W20260928 06:49:34.412415 7360 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:49:34 dist-test-slave-twl6 krb5kdc[29213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:49:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:49:34.533138 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:34239
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:43139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:34239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:34.641161 29226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:34.641412 29226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:34.641450 29226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:34.645417 29226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:34.645473 29226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:34.645493 29226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:34.645551 29226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:34.645586 29226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:34.650686 29226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:43139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:34239
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:34239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:34.651681 29226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:34.652693 29226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:34.661543 29234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:34.661598 29226 server_base.cc:1034] running on GCE node
W20260928 06:49:34.661545 29232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:34.661556 29231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:34.662348 29226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:34.663409 29226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:34.664634 29226 hybrid_clock.cc:648] HybridClock initialized: now 1790578174664612 us; error 35 us; skew 500 ppm
Sep 28 06:49:34 dist-test-slave-twl6 krb5kdc[29213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578174, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:34.667241 29226 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:49:34.668205 29226 webserver.cc:460] Webserver started at http://127.7.48.62:43407/ using document root <none> and password file <none>
I20260928 06:49:34.668759 29226 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:34.668846 29226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:34.669070 29226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:34.671029 29226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "988d8701d0d84d058f8185c7d6243933"
format_stamp: "Formatted at 2026-09-28 06:49:34 on dist-test-slave-twl6"
server_key: "73eeccba1d11407482104379f811decc"
server_key_iv: "ebe8c483a9b21925381cc441439a8518"
server_key_version: "encryptionkey@0"
I20260928 06:49:34.671581 29226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "988d8701d0d84d058f8185c7d6243933"
format_stamp: "Formatted at 2026-09-28 06:49:34 on dist-test-slave-twl6"
server_key: "73eeccba1d11407482104379f811decc"
server_key_iv: "ebe8c483a9b21925381cc441439a8518"
server_key_version: "encryptionkey@0"
I20260928 06:49:34.675256 29226 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 06:49:34.677698 29241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:34.678918 29226 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 06:49:34.679075 29226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "988d8701d0d84d058f8185c7d6243933"
format_stamp: "Formatted at 2026-09-28 06:49:34 on dist-test-slave-twl6"
server_key: "73eeccba1d11407482104379f811decc"
server_key_iv: "ebe8c483a9b21925381cc441439a8518"
server_key_version: "encryptionkey@0"
I20260928 06:49:34.679177 29226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:34.683497 29226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:34.686342 29226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:34.686538 29226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:34.693173 29226 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:34239
I20260928 06:49:34.693260 29293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:34239 every 8 connection(s)
I20260928 06:49:34.694140 29226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:34.696882 29294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:34.699055 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 29226
I20260928 06:49:34.699205 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:34.699474 7360 external_mini_cluster.cc:1428] Setting key 59c4e690373b6a5ea83a6953d23bf4e6
I20260928 06:49:34.702960 29294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 988d8701d0d84d058f8185c7d6243933: Bootstrap starting.
I20260928 06:49:34.705250 29297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:34.700927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48507 (local address 127.7.48.62:34239)
0928 06:49:34.701328 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:34.701350 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:34.701391 (+ 41us) server_negotiation.cc:407] Connection header received
0928 06:49:34.702218 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:34.702242 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:34.702608 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:34.702960 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:34.703819 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:34.703825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:34.703839 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:34.703851 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:34.703966 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:34.703997 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:34.704141 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:34.704217 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 06:49:34.704368 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":99,"threads_started":1}
I20260928 06:49:34.705689 29294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 988d8701d0d84d058f8185c7d6243933: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:34.706584 29294 log.cc:824] T 00000000000000000000000000000000 P 988d8701d0d84d058f8185c7d6243933: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:34.708237 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:49:34.708884 29294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 988d8701d0d84d058f8185c7d6243933: No bootstrap required, opened a new log
I20260928 06:49:34.712131 29297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:34.710419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33233 (local address 127.7.48.62:34239)
0928 06:49:34.710544 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:34.710548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:34.710795 (+ 247us) server_negotiation.cc:407] Connection header received
0928 06:49:34.710906 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:34.710908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:34.710936 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:34.710998 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:34.711659 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:34.711661 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:34.711663 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:34.711672 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:34.711694 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:34.711704 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:34.711767 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:34.711916 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 06:49:34.711957 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:49:34.713384 29294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 988d8701d0d84d058f8185c7d6243933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "988d8701d0d84d058f8185c7d6243933" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34239 } }
I20260928 06:49:34.713653 29294 raft_consensus.cc:374] T 00000000000000000000000000000000 P 988d8701d0d84d058f8185c7d6243933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:34.713713 29294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 988d8701d0d84d058f8185c7d6243933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 988d8701d0d84d058f8185c7d6243933, State: Initialized, Role: FOLLOWER
I20260928 06:49:34.714109 29294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 988d8701d0d84d058f8185c7d6243933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "988d8701d0d84d058f8185c7d6243933" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34239 } }
I20260928 06:49:34.714259 29294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 988d8701d0d84d058f8185c7d6243933 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:49:34.714329 29294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 988d8701d0d84d058f8185c7d6243933 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:49:34.714433 29294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 988d8701d0d84d058f8185c7d6243933 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:34.715332 29294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 988d8701d0d84d058f8185c7d6243933 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "988d8701d0d84d058f8185c7d6243933" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34239 } }
I20260928 06:49:34.715680 29294 leader_election.cc:302] T 00000000000000000000000000000000 P 988d8701d0d84d058f8185c7d6243933 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 988d8701d0d84d058f8185c7d6243933; no voters:
I20260928 06:49:34.716037 29294 leader_election.cc:288] T 00000000000000000000000000000000 P 988d8701d0d84d058f8185c7d6243933 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:49:34.716168 29304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 988d8701d0d84d058f8185c7d6243933 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:34.716415 29304 raft_consensus.cc:686] T 00000000000000000000000000000000 P 988d8701d0d84d058f8185c7d6243933 [term 1 LEADER]: Becoming Leader. State: Replica: 988d8701d0d84d058f8185c7d6243933, State: Running, Role: LEADER
I20260928 06:49:34.716768 29304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 988d8701d0d84d058f8185c7d6243933 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "988d8701d0d84d058f8185c7d6243933" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34239 } }
I20260928 06:49:34.717255 29294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 988d8701d0d84d058f8185c7d6243933 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:34.718570 29306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 988d8701d0d84d058f8185c7d6243933 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 988d8701d0d84d058f8185c7d6243933. Latest consensus state: current_term: 1 leader_uuid: "988d8701d0d84d058f8185c7d6243933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "988d8701d0d84d058f8185c7d6243933" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34239 } } }
I20260928 06:49:34.718585 29305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 988d8701d0d84d058f8185c7d6243933 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "988d8701d0d84d058f8185c7d6243933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "988d8701d0d84d058f8185c7d6243933" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34239 } } }
I20260928 06:49:34.718699 29306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 988d8701d0d84d058f8185c7d6243933 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:34.718700 29305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 988d8701d0d84d058f8185c7d6243933 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:34.719107 29313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:34.722002 29313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:49:34.726934 29313 catalog_manager.cc:1347] Generated new cluster ID: 8d761bc64fb34c9eb3a051a19453584c
I20260928 06:49:34.727003 29313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:34.736357 29313 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:34.736661 29313 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:34.743824 29313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 988d8701d0d84d058f8185c7d6243933: Generated new TSK 0
I20260928 06:49:34.744645 29313 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:49:34.749083 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 29226
2026-09-28T06:49:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:49:34 dist-test-slave-twl6 krb5kdc[29325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:49:34 dist-test-slave-twl6 krb5kdc[29325](info): set up 2 sockets
Sep 28 06:49:34 dist-test-slave-twl6 krb5kdc[29325](info): commencing operation
krb5kdc: starting...
W20260928 06:49:35.748373 7360 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:49:35 dist-test-slave-twl6 krb5kdc[29325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:49:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:49:35.862313 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:34361
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:45793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:34361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:35.963954 29338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:35.964210 29338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:35.964277 29338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:35.968163 29338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:35.968231 29338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:35.968266 29338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:35.968318 29338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:35.968354 29338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:35.972867 29338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:45793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:34361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:34361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:35.973933 29338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:35.974926 29338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:35.982828 29346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:35.982928 29338 server_base.cc:1034] running on GCE node
W20260928 06:49:35.982828 29343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:35.982851 29344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:35.983563 29338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:35.984678 29338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:35.985864 29338 hybrid_clock.cc:648] HybridClock initialized: now 1790578175985841 us; error 36 us; skew 500 ppm
Sep 28 06:49:35 dist-test-slave-twl6 krb5kdc[29325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578175, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:35.988355 29338 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:49:35.989256 29338 webserver.cc:460] Webserver started at http://127.7.48.62:34165/ using document root <none> and password file <none>
I20260928 06:49:35.989758 29338 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:35.989811 29338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:35.990051 29338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:35.991976 29338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "fe134a4a9c444883821bca9c767d4d65"
format_stamp: "Formatted at 2026-09-28 06:49:35 on dist-test-slave-twl6"
server_key: "59cbcfa2086e58bb0dbc4b628c9a4886"
server_key_iv: "56eb81cdb039c66516d9d16ed2d1b82c"
server_key_version: "encryptionkey@0"
I20260928 06:49:35.992515 29338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "fe134a4a9c444883821bca9c767d4d65"
format_stamp: "Formatted at 2026-09-28 06:49:35 on dist-test-slave-twl6"
server_key: "59cbcfa2086e58bb0dbc4b628c9a4886"
server_key_iv: "56eb81cdb039c66516d9d16ed2d1b82c"
server_key_version: "encryptionkey@0"
I20260928 06:49:35.995980 29338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260928 06:49:35.998090 29353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:35.999147 29338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:49:35.999274 29338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "fe134a4a9c444883821bca9c767d4d65"
format_stamp: "Formatted at 2026-09-28 06:49:35 on dist-test-slave-twl6"
server_key: "59cbcfa2086e58bb0dbc4b628c9a4886"
server_key_iv: "56eb81cdb039c66516d9d16ed2d1b82c"
server_key_version: "encryptionkey@0"
I20260928 06:49:35.999370 29338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:36.013679 29338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:36.016206 29338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:36.016372 29338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:36.022653 29338 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:34361
I20260928 06:49:36.022680 29405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:34361 every 8 connection(s)
I20260928 06:49:36.023483 29338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:36.026041 29406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:36.028008 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 29338
I20260928 06:49:36.028146 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:36.028445 7360 external_mini_cluster.cc:1428] Setting key 73e1e5882244729127966148a6b062ac
I20260928 06:49:36.032275 29406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe134a4a9c444883821bca9c767d4d65: Bootstrap starting.
I20260928 06:49:36.034324 29406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe134a4a9c444883821bca9c767d4d65: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:36.035084 29406 log.cc:824] T 00000000000000000000000000000000 P fe134a4a9c444883821bca9c767d4d65: Log is configured to *not* fsync() on all Append() calls
Sep 28 06:49:36 dist-test-slave-twl6 krb5kdc[29325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:49:36.037029 29406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe134a4a9c444883821bca9c767d4d65: No bootstrap required, opened a new log
I20260928 06:49:36.041399 29406 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe134a4a9c444883821bca9c767d4d65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe134a4a9c444883821bca9c767d4d65" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34361 } }
I20260928 06:49:36.041643 29406 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe134a4a9c444883821bca9c767d4d65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:36.041698 29406 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe134a4a9c444883821bca9c767d4d65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe134a4a9c444883821bca9c767d4d65, State: Initialized, Role: FOLLOWER
I20260928 06:49:36.041915 29409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:36.029785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38211 (local address 127.7.48.62:34361)
0928 06:49:36.030186 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:36.030199 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:36.030230 (+ 31us) server_negotiation.cc:407] Connection header received
0928 06:49:36.030806 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:36.030822 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:36.031123 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:36.031427 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:36.032265 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:36.033201 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:36.034020 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:36.034521 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:36.034729 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:36.037237 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:36.037244 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:36.037259 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:36.037287 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:36.039097 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:36.039490 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:36.039495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:36.039500 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:36.039633 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:36.039865 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:36.039868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:36.039870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:36.040357 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:36.040539 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:36.040760 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 06:49:36.041047 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":124,"threads_started":1}
I20260928 06:49:36.042122 29406 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe134a4a9c444883821bca9c767d4d65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe134a4a9c444883821bca9c767d4d65" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34361 } }
I20260928 06:49:36.042264 29406 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe134a4a9c444883821bca9c767d4d65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:49:36.042321 29406 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe134a4a9c444883821bca9c767d4d65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:49:36.042407 29406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe134a4a9c444883821bca9c767d4d65 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:36.043287 29406 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe134a4a9c444883821bca9c767d4d65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe134a4a9c444883821bca9c767d4d65" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34361 } }
I20260928 06:49:36.043591 29406 leader_election.cc:302] T 00000000000000000000000000000000 P fe134a4a9c444883821bca9c767d4d65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe134a4a9c444883821bca9c767d4d65; no voters:
I20260928 06:49:36.043933 29406 leader_election.cc:288] T 00000000000000000000000000000000 P fe134a4a9c444883821bca9c767d4d65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:49:36.044008 29411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe134a4a9c444883821bca9c767d4d65 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:36.044186 29411 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe134a4a9c444883821bca9c767d4d65 [term 1 LEADER]: Becoming Leader. State: Replica: fe134a4a9c444883821bca9c767d4d65, State: Running, Role: LEADER
I20260928 06:49:36.044494 29411 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe134a4a9c444883821bca9c767d4d65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe134a4a9c444883821bca9c767d4d65" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34361 } }
I20260928 06:49:36.044597 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:49:36.045135 29406 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe134a4a9c444883821bca9c767d4d65 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:36.045701 29411 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe134a4a9c444883821bca9c767d4d65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe134a4a9c444883821bca9c767d4d65. Latest consensus state: current_term: 1 leader_uuid: "fe134a4a9c444883821bca9c767d4d65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe134a4a9c444883821bca9c767d4d65" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34361 } } }
I20260928 06:49:36.045897 29411 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe134a4a9c444883821bca9c767d4d65 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:36.046105 29413 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe134a4a9c444883821bca9c767d4d65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe134a4a9c444883821bca9c767d4d65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe134a4a9c444883821bca9c767d4d65" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 34361 } } }
I20260928 06:49:36.046254 29413 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe134a4a9c444883821bca9c767d4d65 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:36.046314 29423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:36.049557 29423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:49:36.055487 29409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:36.046708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57997 (local address 127.7.48.62:34361)
0928 06:49:36.046832 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:36.046836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:36.047226 (+ 390us) server_negotiation.cc:407] Connection header received
0928 06:49:36.047409 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:36.047415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:36.047464 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:36.047518 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:36.048432 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:36.049170 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:36.050290 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:36.050517 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:36.050629 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:36.052095 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:36.052098 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:36.052101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:36.052125 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:36.053755 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:36.054246 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:36.054250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:36.054252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:36.054342 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:36.054663 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:36.054667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:36.054668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:36.054841 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:36.054987 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:36.055217 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 06:49:36.055330 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:49:36.055866 29423 catalog_manager.cc:1347] Generated new cluster ID: 2a8f1f4be5c94bc78758dca4de665b7a
I20260928 06:49:36.055944 29423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:36.063849 29423 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:36.064673 29423 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:36.070322 29423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe134a4a9c444883821bca9c767d4d65: Generated new TSK 0
I20260928 06:49:36.070928 29423 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:49:36.080505 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 29338
2026-09-28T06:49:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:49:36 dist-test-slave-twl6 krb5kdc[29437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:49:36 dist-test-slave-twl6 krb5kdc[29437](info): set up 2 sockets
Sep 28 06:49:36 dist-test-slave-twl6 krb5kdc[29437](info): commencing operation
krb5kdc: starting...
W20260928 06:49:37.044492 7360 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:49:37 dist-test-slave-twl6 krb5kdc[29437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:49:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:49:37.157064 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:33723
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:43761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:33723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:37.264539 29450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:37.264791 29450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:37.264832 29450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:37.268733 29450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:37.268790 29450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:37.268810 29450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:37.268826 29450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:37.268842 29450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:37.273869 29450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:43761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:33723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:33723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:37.274837 29450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:37.275779 29450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:37.284013 29458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:37.284155 29450 server_base.cc:1034] running on GCE node
W20260928 06:49:37.284014 29456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:37.284068 29455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:37.284951 29450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:37.286044 29450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:37.287245 29450 hybrid_clock.cc:648] HybridClock initialized: now 1790578177287232 us; error 34 us; skew 500 ppm
Sep 28 06:49:37 dist-test-slave-twl6 krb5kdc[29437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578177, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:37.289963 29450 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:49:37.290776 29450 webserver.cc:460] Webserver started at http://127.7.48.62:41979/ using document root <none> and password file <none>
I20260928 06:49:37.291334 29450 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:37.291431 29450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:37.291661 29450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:37.293716 29450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "a128fb5fad5f4fa6b4f53df430794843"
format_stamp: "Formatted at 2026-09-28 06:49:37 on dist-test-slave-twl6"
server_key: "950ca43df6ac21433def9131dac3140e"
server_key_iv: "119182ae1be7742bf0fc2b7cf0c19298"
server_key_version: "encryptionkey@0"
I20260928 06:49:37.294291 29450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "a128fb5fad5f4fa6b4f53df430794843"
format_stamp: "Formatted at 2026-09-28 06:49:37 on dist-test-slave-twl6"
server_key: "950ca43df6ac21433def9131dac3140e"
server_key_iv: "119182ae1be7742bf0fc2b7cf0c19298"
server_key_version: "encryptionkey@0"
I20260928 06:49:37.297871 29450 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 06:49:37.300230 29465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:37.301491 29450 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 06:49:37.301657 29450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "a128fb5fad5f4fa6b4f53df430794843"
format_stamp: "Formatted at 2026-09-28 06:49:37 on dist-test-slave-twl6"
server_key: "950ca43df6ac21433def9131dac3140e"
server_key_iv: "119182ae1be7742bf0fc2b7cf0c19298"
server_key_version: "encryptionkey@0"
I20260928 06:49:37.301767 29450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:37.332396 29450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:37.335376 29450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:37.335598 29450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:37.342779 29450 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:33723
I20260928 06:49:37.342806 29517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:33723 every 8 connection(s)
I20260928 06:49:37.343729 29450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:37.346370 29518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:37.351651 29518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a128fb5fad5f4fa6b4f53df430794843: Bootstrap starting.
I20260928 06:49:37.352903 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 29450
I20260928 06:49:37.353049 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:37.353348 7360 external_mini_cluster.cc:1428] Setting key bf268e17dc860b6917c5bb1bf0e93e24
I20260928 06:49:37.354539 29518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a128fb5fad5f4fa6b4f53df430794843: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:37.355346 29518 log.cc:824] T 00000000000000000000000000000000 P a128fb5fad5f4fa6b4f53df430794843: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:37.357215 29518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a128fb5fad5f4fa6b4f53df430794843: No bootstrap required, opened a new log
I20260928 06:49:37.360565 29518 raft_consensus.cc:348] T 00000000000000000000000000000000 P a128fb5fad5f4fa6b4f53df430794843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a128fb5fad5f4fa6b4f53df430794843" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33723 } }
I20260928 06:49:37.360766 29518 raft_consensus.cc:374] T 00000000000000000000000000000000 P a128fb5fad5f4fa6b4f53df430794843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:37.360818 29518 raft_consensus.cc:729] T 00000000000000000000000000000000 P a128fb5fad5f4fa6b4f53df430794843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a128fb5fad5f4fa6b4f53df430794843, State: Initialized, Role: FOLLOWER
I20260928 06:49:37.361310 29518 consensus_queue.cc:260] T 00000000000000000000000000000000 P a128fb5fad5f4fa6b4f53df430794843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a128fb5fad5f4fa6b4f53df430794843" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33723 } }
I20260928 06:49:37.361498 29518 raft_consensus.cc:388] T 00000000000000000000000000000000 P a128fb5fad5f4fa6b4f53df430794843 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:49:37.361568 29518 raft_consensus.cc:482] T 00000000000000000000000000000000 P a128fb5fad5f4fa6b4f53df430794843 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:49:37.361688 29518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a128fb5fad5f4fa6b4f53df430794843 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:37.362308 29521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:37.354521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32987 (local address 127.7.48.62:33723)
0928 06:49:37.354984 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:37.355000 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:37.355035 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:49:37.355781 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:37.355804 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:37.356273 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:37.356586 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:37.357003 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:37.357989 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:37.358870 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:37.359351 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:37.359510 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:37.360390 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:37.360402 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:37.360415 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:37.360810 (+ 395us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:37.360898 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:37.360929 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:37.361035 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:37.361196 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 06:49:37.361402 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":159,"threads_started":1}
I20260928 06:49:37.362624 29518 raft_consensus.cc:504] T 00000000000000000000000000000000 P a128fb5fad5f4fa6b4f53df430794843 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a128fb5fad5f4fa6b4f53df430794843" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33723 } }
I20260928 06:49:37.362978 29518 leader_election.cc:302] T 00000000000000000000000000000000 P a128fb5fad5f4fa6b4f53df430794843 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a128fb5fad5f4fa6b4f53df430794843; no voters:
I20260928 06:49:37.363306 29518 leader_election.cc:288] T 00000000000000000000000000000000 P a128fb5fad5f4fa6b4f53df430794843 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:49:37.363404 29523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a128fb5fad5f4fa6b4f53df430794843 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:37.363576 29523 raft_consensus.cc:686] T 00000000000000000000000000000000 P a128fb5fad5f4fa6b4f53df430794843 [term 1 LEADER]: Becoming Leader. State: Replica: a128fb5fad5f4fa6b4f53df430794843, State: Running, Role: LEADER
I20260928 06:49:37.363936 29523 consensus_queue.cc:237] T 00000000000000000000000000000000 P a128fb5fad5f4fa6b4f53df430794843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a128fb5fad5f4fa6b4f53df430794843" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33723 } }
I20260928 06:49:37.364557 29518 sys_catalog.cc:564] T 00000000000000000000000000000000 P a128fb5fad5f4fa6b4f53df430794843 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:37.365249 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:49:37.365559 29524 sys_catalog.cc:455] T 00000000000000000000000000000000 P a128fb5fad5f4fa6b4f53df430794843 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a128fb5fad5f4fa6b4f53df430794843. Latest consensus state: current_term: 1 leader_uuid: "a128fb5fad5f4fa6b4f53df430794843" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a128fb5fad5f4fa6b4f53df430794843" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33723 } } }
I20260928 06:49:37.365688 29524 sys_catalog.cc:458] T 00000000000000000000000000000000 P a128fb5fad5f4fa6b4f53df430794843 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:37.365983 29525 sys_catalog.cc:455] T 00000000000000000000000000000000 P a128fb5fad5f4fa6b4f53df430794843 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a128fb5fad5f4fa6b4f53df430794843" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a128fb5fad5f4fa6b4f53df430794843" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33723 } } }
I20260928 06:49:37.366055 29525 sys_catalog.cc:458] T 00000000000000000000000000000000 P a128fb5fad5f4fa6b4f53df430794843 [sys.catalog]: This master's current role is: LEADER
W20260928 06:49:37.369691 29543 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a128fb5fad5f4fa6b4f53df430794843: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:49:37.369763 29543 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:49:37.369825 29532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:37.370407 29532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:49:37.373791 29521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:37.367621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36613 (local address 127.7.48.62:33723)
0928 06:49:37.367743 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:37.367747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:37.367902 (+ 155us) server_negotiation.cc:407] Connection header received
0928 06:49:37.368043 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:37.368045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:37.368078 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:37.368142 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:37.368461 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:37.369113 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:37.372080 (+ 2967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:37.372398 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:37.372505 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:37.373309 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:37.373313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:37.373315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:37.373327 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:37.373354 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:37.373366 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:37.373453 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:37.373595 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 06:49:37.373643 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:49:37.375451 29532 catalog_manager.cc:1347] Generated new cluster ID: d23c695515e8477e8dd4b5731fbef497
I20260928 06:49:37.375511 29532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:37.388967 29532 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:37.390177 29532 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:37.394210 29532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a128fb5fad5f4fa6b4f53df430794843: Generated new TSK 0
I20260928 06:49:37.394948 29532 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:49:37.400050 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 29450
2026-09-28T06:49:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:49:37 dist-test-slave-twl6 krb5kdc[29549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:49:37 dist-test-slave-twl6 krb5kdc[29549](info): set up 2 sockets
Sep 28 06:49:37 dist-test-slave-twl6 krb5kdc[29549](info): commencing operation
krb5kdc: starting...
W20260928 06:49:38.400396 7360 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:49:38 dist-test-slave-twl6 krb5kdc[29549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:49:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:49:38.519513 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:46047
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:46047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:38.627861 29562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:38.628134 29562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:38.628175 29562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:38.631908 29562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:38.631997 29562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:38.632035 29562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:38.632092 29562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:38.632138 29562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:38.637364 29562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:46047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:46047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:38.638370 29562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:38.639410 29562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:38.647617 29570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:38.647624 29567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:38.647634 29568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:38.647696 29562 server_base.cc:1034] running on GCE node
I20260928 06:49:38.648342 29562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:38.649379 29562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:38.650559 29562 hybrid_clock.cc:648] HybridClock initialized: now 1790578178650542 us; error 30 us; skew 500 ppm
Sep 28 06:49:38 dist-test-slave-twl6 krb5kdc[29549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578178, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:38.658480 29562 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:49:38.659282 29562 webserver.cc:460] Webserver started at http://127.7.48.62:43729/ using document root <none> and password file <none>
I20260928 06:49:38.659793 29562 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:38.659847 29562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:38.660060 29562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:38.662000 29562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "809940347e7e426b8f0f43cbf3d7eca1"
format_stamp: "Formatted at 2026-09-28 06:49:38 on dist-test-slave-twl6"
server_key: "dd8f79d57c5b979a8d614e83a5b8a28e"
server_key_iv: "581307ece4d1dd633e4695ac2e081715"
server_key_version: "encryptionkey@0"
I20260928 06:49:38.662467 29562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "809940347e7e426b8f0f43cbf3d7eca1"
format_stamp: "Formatted at 2026-09-28 06:49:38 on dist-test-slave-twl6"
server_key: "dd8f79d57c5b979a8d614e83a5b8a28e"
server_key_iv: "581307ece4d1dd633e4695ac2e081715"
server_key_version: "encryptionkey@0"
I20260928 06:49:38.665922 29562 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 06:49:38.668135 29577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:38.669265 29562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:49:38.669381 29562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "809940347e7e426b8f0f43cbf3d7eca1"
format_stamp: "Formatted at 2026-09-28 06:49:38 on dist-test-slave-twl6"
server_key: "dd8f79d57c5b979a8d614e83a5b8a28e"
server_key_iv: "581307ece4d1dd633e4695ac2e081715"
server_key_version: "encryptionkey@0"
I20260928 06:49:38.669472 29562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:38.673545 29562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:38.676338 29562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:38.676528 29562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:38.682878 29562 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:46047
I20260928 06:49:38.682888 29629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:46047 every 8 connection(s)
I20260928 06:49:38.683722 29562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:38.685669 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 29562
I20260928 06:49:38.685808 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:38.686138 7360 external_mini_cluster.cc:1428] Setting key f7a553ff5671bdb0a74b64a98f9288a4
I20260928 06:49:38.686753 29630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:38.691120 29633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:38.687480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38705 (local address 127.7.48.62:46047)
0928 06:49:38.687945 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:38.687969 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:38.688003 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:49:38.688620 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:38.688635 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:38.688889 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:38.689219 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:38.689960 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:38.689965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:38.689975 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:38.689985 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:38.690035 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:38.690052 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:38.690200 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:38.690264 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 06:49:38.690407 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":140,"threads_started":1}
I20260928 06:49:38.693065 29630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 809940347e7e426b8f0f43cbf3d7eca1: Bootstrap starting.
I20260928 06:49:38.694084 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:49:38.695443 29630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 809940347e7e426b8f0f43cbf3d7eca1: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:38.696297 29630 log.cc:824] T 00000000000000000000000000000000 P 809940347e7e426b8f0f43cbf3d7eca1: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:38.697754 29633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:38.696178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37667 (local address 127.7.48.62:46047)
0928 06:49:38.696295 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:38.696299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:38.696465 (+ 166us) server_negotiation.cc:407] Connection header received
0928 06:49:38.696593 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:38.696594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:38.696619 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:38.696701 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:38.697372 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:38.697375 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:38.697378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:49:38.697389 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:38.697414 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:49:38.697427 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:38.697518 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:38.697596 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 06:49:38.697642 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 06:49:38.698186 29630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 809940347e7e426b8f0f43cbf3d7eca1: No bootstrap required, opened a new log
I20260928 06:49:38.701465 29630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 809940347e7e426b8f0f43cbf3d7eca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809940347e7e426b8f0f43cbf3d7eca1" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 46047 } }
I20260928 06:49:38.701699 29630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 809940347e7e426b8f0f43cbf3d7eca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:49:38.701776 29630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 809940347e7e426b8f0f43cbf3d7eca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809940347e7e426b8f0f43cbf3d7eca1, State: Initialized, Role: FOLLOWER
I20260928 06:49:38.702204 29630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 809940347e7e426b8f0f43cbf3d7eca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809940347e7e426b8f0f43cbf3d7eca1" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 46047 } }
I20260928 06:49:38.702374 29630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 809940347e7e426b8f0f43cbf3d7eca1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:49:38.702461 29630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 809940347e7e426b8f0f43cbf3d7eca1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:49:38.702557 29630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 809940347e7e426b8f0f43cbf3d7eca1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:38.703401 29630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 809940347e7e426b8f0f43cbf3d7eca1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809940347e7e426b8f0f43cbf3d7eca1" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 46047 } }
I20260928 06:49:38.703737 29630 leader_election.cc:302] T 00000000000000000000000000000000 P 809940347e7e426b8f0f43cbf3d7eca1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 809940347e7e426b8f0f43cbf3d7eca1; no voters:
I20260928 06:49:38.704100 29630 leader_election.cc:288] T 00000000000000000000000000000000 P 809940347e7e426b8f0f43cbf3d7eca1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:49:38.704176 29640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 809940347e7e426b8f0f43cbf3d7eca1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:38.704427 29640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 809940347e7e426b8f0f43cbf3d7eca1 [term 1 LEADER]: Becoming Leader. State: Replica: 809940347e7e426b8f0f43cbf3d7eca1, State: Running, Role: LEADER
I20260928 06:49:38.704735 29640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 809940347e7e426b8f0f43cbf3d7eca1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809940347e7e426b8f0f43cbf3d7eca1" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 46047 } }
I20260928 06:49:38.705396 29630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 809940347e7e426b8f0f43cbf3d7eca1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:38.706506 29641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 809940347e7e426b8f0f43cbf3d7eca1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "809940347e7e426b8f0f43cbf3d7eca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809940347e7e426b8f0f43cbf3d7eca1" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 46047 } } }
I20260928 06:49:38.706516 29642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 809940347e7e426b8f0f43cbf3d7eca1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 809940347e7e426b8f0f43cbf3d7eca1. Latest consensus state: current_term: 1 leader_uuid: "809940347e7e426b8f0f43cbf3d7eca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809940347e7e426b8f0f43cbf3d7eca1" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 46047 } } }
I20260928 06:49:38.706655 29641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 809940347e7e426b8f0f43cbf3d7eca1 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:38.706660 29642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 809940347e7e426b8f0f43cbf3d7eca1 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:38.707053 29649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:38.709699 29649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:49:38.714648 29649 catalog_manager.cc:1347] Generated new cluster ID: 869d98e8993e4460afc0cb0d646ac679
I20260928 06:49:38.714711 29649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:38.738619 29649 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:38.738956 29649 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:38.743575 29649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 809940347e7e426b8f0f43cbf3d7eca1: Generated new TSK 0
I20260928 06:49:38.744379 29649 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:49:38.755261 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 29562
2026-09-28T06:49:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1356 ms)
[----------] 18 tests from AuthTokenIssuingTest (24470 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:49:38 dist-test-slave-twl6 krb5kdc[29661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:49:38 dist-test-slave-twl6 krb5kdc[29661](info): set up 2 sockets
Sep 28 06:49:38 dist-test-slave-twl6 krb5kdc[29661](info): commencing operation
krb5kdc: starting...
W20260928 06:49:39.752341 7360 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:49:39 dist-test-slave-twl6 krb5kdc[29661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:49:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:49:39.875209 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:35745
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:35745,127.7.48.61:38121,127.7.48.60:46753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:39.997438 29674 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:49:39.997716 29674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:39.997740 29674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:39.997777 29674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:40.001502 29674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:40.001549 29674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:40.001566 29674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:40.001582 29674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:40.001596 29674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:40.006389 29674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:35745,127.7.48.61:38121,127.7.48.60:46753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:35745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:40.007314 29674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:40.008249 29674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:40.016208 29679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:40.016187 29682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:40.016191 29680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:40.016263 29674 server_base.cc:1034] running on GCE node
I20260928 06:49:40.017002 29674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:40.018002 29674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:40.019155 29674 hybrid_clock.cc:648] HybridClock initialized: now 1790578180019142 us; error 24 us; skew 500 ppm
Sep 28 06:49:40 dist-test-slave-twl6 krb5kdc[29661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578180, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:40.021778 29674 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:49:40.022658 29674 webserver.cc:460] Webserver started at http://127.7.48.62:35581/ using document root <none> and password file <none>
I20260928 06:49:40.023192 29674 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:40.023249 29674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:40.023473 29674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:40.025489 29674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "66c51594010848cbbc466d5cbadd3630"
format_stamp: "Formatted at 2026-09-28 06:49:40 on dist-test-slave-twl6"
server_key: "74c8faa1947ce9d854b43ea4d7620284"
server_key_iv: "5d82420a4be7be705b5f443a2329be5b"
server_key_version: "encryptionkey@0"
I20260928 06:49:40.026039 29674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "66c51594010848cbbc466d5cbadd3630"
format_stamp: "Formatted at 2026-09-28 06:49:40 on dist-test-slave-twl6"
server_key: "74c8faa1947ce9d854b43ea4d7620284"
server_key_iv: "5d82420a4be7be705b5f443a2329be5b"
server_key_version: "encryptionkey@0"
I20260928 06:49:40.029536 29674 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260928 06:49:40.031734 29689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:40.032852 29674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 06:49:40.032987 29674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "66c51594010848cbbc466d5cbadd3630"
format_stamp: "Formatted at 2026-09-28 06:49:40 on dist-test-slave-twl6"
server_key: "74c8faa1947ce9d854b43ea4d7620284"
server_key_iv: "5d82420a4be7be705b5f443a2329be5b"
server_key_version: "encryptionkey@0"
I20260928 06:49:40.033090 29674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:40.046434 29674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:40.049505 29674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:40.049661 29674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:40.056417 29674 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:35745
I20260928 06:49:40.056722 29741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:35745 every 8 connection(s)
I20260928 06:49:40.057379 29674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:40.060490 29742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:40.062260 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 29674
I20260928 06:49:40.062427 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:40.062714 7360 external_mini_cluster.cc:1428] Setting key 5ee2d08bbe56c3f27e9e148efd4828ae
I20260928 06:49:40.064213 29742 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.62" port: 35745 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.48.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.61@KRBTEST.COM" created.
Sep 28 06:49:40 dist-test-slave-twl6 krb5kdc[29661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578180, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:49:40.079682 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.066466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:35745 (local address 127.0.0.1:54854)
0928 06:49:40.067006 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.067039 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:40.067874 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:40.068600 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:40.068605 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:40.069052 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:40.069306 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:40.069319 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.070693 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.070696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:40.071422 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:40.071431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.072188 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.072192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:40.072312 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:40.072960 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:40.072985 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:40.074699 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:40.076769 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:40.076779 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:40.076789 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:40.077076 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:40.077385 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:40.077389 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:40.077391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:40.077497 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:40.078037 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:40.078043 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:40.078186 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 06:49:40.078348 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 06:49:40.078716 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":108,"thread_start_us":153,"threads_started":1}
I20260928 06:49:40.079682 29745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.066465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54854 (local address 127.7.48.62:35745)
0928 06:49:40.066865 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.066882 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.067149 (+ 267us) server_negotiation.cc:407] Connection header received
0928 06:49:40.068238 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.068242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.068415 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.068605 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.069457 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.070569 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.071544 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.072025 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.072195 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:40.074862 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:40.074870 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:40.074884 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:40.074913 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:40.076663 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.077182 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.077187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.077189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.077289 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.077582 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.077586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.077588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.077928 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:40.078064 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:40.078400 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 06:49:40.078646 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":133,"threads_started":1}
I20260928 06:49:40.081967 29742 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.61" port: 38121 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:49:40.082652 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.082405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:38121 (local address 127.0.0.1:50870)
0928 06:49:40.082523 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.082595 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.61:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 06:49:40.082841 29690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.48.61:38121: connect: Connection refused (error 111)
I20260928 06:49:40.083660 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.083400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:38121 (local address 127.0.0.1:50880)
0928 06:49:40.083529 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.083605 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.61:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 06:49:40.083806 29742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.61:38121: Network error: Client connection negotiation failed: client connection to 127.7.48.61:38121: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.61.keytab.
Entry for principal kudu/127.7.48.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.61.keytab.
I20260928 06:49:40.086691 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.61:38121
--webserver_interface=127.7.48.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.61.keytab
--principal=kudu/127.7.48.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:35745,127.7.48.61:38121,127.7.48.60:46753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:49:40.133073 29742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.61" port: 38121 } attempt: 1
I20260928 06:49:40.134027 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.133725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:38121 (local address 127.0.0.1:50888)
0928 06:49:40.133885 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.133951 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.61:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 06:49:40.134743 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.134522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:38121 (local address 127.0.0.1:50890)
0928 06:49:40.134646 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.134694 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.61:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 06:49:40.134861 29742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.61:38121: Network error: Client connection negotiation failed: client connection to 127.7.48.61:38121: connect: Connection refused (error 111)
W20260928 06:49:40.195654 29749 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:49:40.196080 29749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:40.196177 29749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:40.196252 29749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:40.200198 29749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:40.200305 29749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:40.200340 29749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:40.200380 29749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:40.200414 29749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:40.203023 29742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.61" port: 38121 } attempt: 2
I20260928 06:49:40.203859 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.203556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:38121 (local address 127.0.0.1:50904)
0928 06:49:40.203713 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.203783 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.61:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 06:49:40.204627 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.204399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:38121 (local address 127.0.0.1:50918)
0928 06:49:40.204515 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.204562 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.61:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 06:49:40.204741 29742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.61:38121: Network error: Client connection negotiation failed: client connection to 127.7.48.61:38121: connect: Connection refused (error 111)
I20260928 06:49:40.206252 29749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:35745,127.7.48.61:38121,127.7.48.60:46753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.61.keytab
--principal=kudu/127.7.48.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.61:38121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.48.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:40.207454 29749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:40.208582 29749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:40.218425 29755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:40.218616 29749 server_base.cc:1034] running on GCE node
W20260928 06:49:40.218425 29757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:40.218437 29754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:40.219340 29749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:40.220551 29749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:40.221753 29749 hybrid_clock.cc:648] HybridClock initialized: now 1790578180221738 us; error 29 us; skew 500 ppm
Sep 28 06:49:40 dist-test-slave-twl6 krb5kdc[29661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578180, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:40.224680 29749 init.cc:377] Logged in from keytab as kudu/127.7.48.61@KRBTEST.COM (short username kudu)
I20260928 06:49:40.225644 29749 webserver.cc:460] Webserver started at http://127.7.48.61:35425/ using document root <none> and password file <none>
I20260928 06:49:40.226233 29749 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:40.226329 29749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:40.226568 29749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:40.228803 29749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-1/data/instance:
uuid: "676cf99ebbea4928b12aec4792fe64b6"
format_stamp: "Formatted at 2026-09-28 06:49:40 on dist-test-slave-twl6"
server_key: "a4c68ab8d0e2ee2fc8c50f2b20159b35"
server_key_iv: "6283f926d1cde0910a9b37b918892ea5"
server_key_version: "encryptionkey@0"
I20260928 06:49:40.229393 29749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-1/wal/instance:
uuid: "676cf99ebbea4928b12aec4792fe64b6"
format_stamp: "Formatted at 2026-09-28 06:49:40 on dist-test-slave-twl6"
server_key: "a4c68ab8d0e2ee2fc8c50f2b20159b35"
server_key_iv: "6283f926d1cde0910a9b37b918892ea5"
server_key_version: "encryptionkey@0"
I20260928 06:49:40.233296 29749 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 06:49:40.235769 29764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:40.237110 29749 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:49:40.237258 29749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-1/wal
uuid: "676cf99ebbea4928b12aec4792fe64b6"
format_stamp: "Formatted at 2026-09-28 06:49:40 on dist-test-slave-twl6"
server_key: "a4c68ab8d0e2ee2fc8c50f2b20159b35"
server_key_iv: "6283f926d1cde0910a9b37b918892ea5"
server_key_version: "encryptionkey@0"
I20260928 06:49:40.237375 29749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:40.273722 29749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:40.276893 29749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:40.277113 29749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:40.284780 29749 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.61:38121
I20260928 06:49:40.285090 29816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.61:38121 every 8 connection(s)
I20260928 06:49:40.285859 29749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-1/data/info.pb
I20260928 06:49:40.288722 29817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:40.292268 29817 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.62" port: 35745 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:49:40.293761 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 29749
I20260928 06:49:40.293892 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-1/wal/instance
I20260928 06:49:40.294219 7360 external_mini_cluster.cc:1428] Setting key 8eeca092fac8c405e2ef25010a3fb11f
I20260928 06:49:40.295943 29742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.61" port: 38121 } attempt: 3
Sep 28 06:49:40 dist-test-slave-twl6 krb5kdc[29661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578180, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for kudu/127.7.48.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.48.60@KRBTEST.COM; defaulting to no policy
Sep 28 06:49:40 dist-test-slave-twl6 krb5kdc[29661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578180, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.61@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.60@KRBTEST.COM" created.
I20260928 06:49:40.307596 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.296412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:38121 (local address 127.0.0.1:50924)
0928 06:49:40.296597 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.296612 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:40.296697 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:40.297412 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:40.297418 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:40.297638 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:40.297800 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:40.297810 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.299141 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.299144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:40.300138 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:40.300153 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.301043 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.301048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:40.301157 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:40.301726 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:40.301740 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:40.303032 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:40.305634 (+ 2602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:40.305637 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:40.305638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:40.305875 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:40.306314 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:40.306320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:40.306323 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:40.306400 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:40.307108 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:40.307111 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:40.307205 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 06:49:40.307307 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 06:49:40.307437 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260928 06:49:40.308437 29745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.294254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54870 (local address 127.7.48.62:35745)
0928 06:49:40.294427 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.294434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.295379 (+ 945us) server_negotiation.cc:407] Connection header received
0928 06:49:40.296522 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.296525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.296596 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.296659 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.298262 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.299139 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.300181 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.300544 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.300667 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:40.303945 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:40.303949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:40.303951 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:40.303980 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:40.305717 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.306257 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.306263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.306266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.306356 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.306729 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.306732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.306734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.306908 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:40.307011 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:40.308242 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0928 06:49:40.308333 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"spinlock_wait_cycles":7424}
I20260928 06:49:40.308629 29821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.296493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50924 (local address 127.7.48.61:38121)
0928 06:49:40.296912 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.296923 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.296942 (+ 19us) server_negotiation.cc:407] Connection header received
0928 06:49:40.297024 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.297028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.297231 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.297417 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.297940 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.299026 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.300262 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.300829 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.300948 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:40.303195 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:40.303205 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:40.303223 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:40.303254 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:40.305503 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.306000 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.306009 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.306014 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.306170 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.306514 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.306516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.306517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.306970 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:40.307148 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:40.307411 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 06:49:40.307681 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":329,"thread_start_us":97,"threads_started":1}
I20260928 06:49:40.308946 29819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.294661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:35745 (local address 127.0.0.1:54870)
0928 06:49:40.295172 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.295251 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:40.296261 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:40.297051 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:40.297059 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:40.297545 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:40.298100 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:40.298113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.299264 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.299267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:40.300052 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:40.300061 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.300723 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.300728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:40.300908 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:40.301599 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:40.301626 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:40.303740 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:40.305845 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:40.305855 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:40.305865 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:40.306142 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:40.306465 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:40.306468 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:40.306470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:40.306624 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:40.307019 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:40.307027 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:40.307187 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 06:49:40.308600 (+ 1413us) client_negotiation.cc:240] Negotiation successful
0928 06:49:40.308810 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":25,"spinlock_wait_cycles":3840,"thread_start_us":151,"threads_started":1}
I20260928 06:49:40.310863 29817 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.61" port: 38121 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:49:40.311231 29742 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:49:40.311952 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.311661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:46753 (local address 127.0.0.1:38050)
0928 06:49:40.311805 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.311844 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 06:49:40.312809 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.312453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:46753 (local address 127.0.0.1:38056)
0928 06:49:40.312662 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.312735 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260928 06:49:40.313052 29742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.60:46753: Network error: Client connection negotiation failed: client connection to 127.7.48.60:46753: connect: Connection refused (error 111)
Sep 28 06:49:40 dist-test-slave-twl6 krb5kdc[29661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578180, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.61@KRBTEST.COM for kudu/127.7.48.61@KRBTEST.COM
I20260928 06:49:40.319608 29819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.311407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:38121 (local address 127.0.0.1:50936)
0928 06:49:40.311524 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.311540 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:40.311615 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:40.312095 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:40.312101 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:40.312296 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:40.312394 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:40.312404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.313251 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.313256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:40.314216 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:40.314230 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.314661 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.314668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:40.314781 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:40.315204 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:40.315219 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:40.316534 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:40.318190 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:40.318194 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:40.318196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:40.318378 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:40.318646 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:40.318649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:40.318650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:40.318737 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:40.319116 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:40.319119 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:40.319194 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 06:49:40.319295 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 06:49:40.319438 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 06:49:40.319610 29821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.311627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50936 (local address 127.7.48.61:38121)
0928 06:49:40.311743 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.311749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.311764 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:49:40.311801 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.311806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.311857 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.311977 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.312528 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.313135 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.314331 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.314548 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.314628 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:40.316680 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:40.316683 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:40.316685 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:40.316704 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:40.318059 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.318470 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.318475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.318477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.318548 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.318854 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.318857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.318858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.319023 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:40.319123 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:40.319347 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 06:49:40.319493 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":36}
I20260928 06:49:40.320557 29817 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:49:40.321237 29819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.320974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:46753 (local address 127.0.0.1:38060)
0928 06:49:40.321093 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.321183 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 06:49:40.321456 29768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.48.60:46753: connect: Connection refused (error 111)
I20260928 06:49:40.322364 29819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.322058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:46753 (local address 127.0.0.1:38064)
0928 06:49:40.322263 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.322317 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260928 06:49:40.322521 29817 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.60:46753: Network error: Client connection negotiation failed: client connection to 127.7.48.60:46753: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.60.keytab.
Entry for principal kudu/127.7.48.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.60.keytab.
I20260928 06:49:40.324564 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.60:46753
--webserver_interface=127.7.48.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.60.keytab
--principal=kudu/127.7.48.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:35745,127.7.48.61:38121,127.7.48.60:46753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:49:40.344218 29742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } attempt: 1
I20260928 06:49:40.345088 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.344783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:46753 (local address 127.0.0.1:38066)
0928 06:49:40.344943 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.345005 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 06:49:40.345816 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.345610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:46753 (local address 127.0.0.1:38072)
0928 06:49:40.345716 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.345767 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 06:49:40.345930 29742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.60:46753: Network error: Client connection negotiation failed: client connection to 127.7.48.60:46753: connect: Connection refused (error 111)
I20260928 06:49:40.371944 29817 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } attempt: 1
I20260928 06:49:40.372846 29819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.372530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:46753 (local address 127.0.0.1:38084)
0928 06:49:40.372701 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.372770 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 06:49:40.373974 29819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.373760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:46753 (local address 127.0.0.1:38092)
0928 06:49:40.373871 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.373923 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 06:49:40.374107 29817 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.60:46753: Network error: Client connection negotiation failed: client connection to 127.7.48.60:46753: connect: Connection refused (error 111)
I20260928 06:49:40.421098 29742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } attempt: 2
I20260928 06:49:40.422063 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.421658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:46753 (local address 127.0.0.1:38094)
0928 06:49:40.421845 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.421944 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 06:49:40.423148 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.422676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:46753 (local address 127.0.0.1:38098)
0928 06:49:40.422929 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.423062 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
W20260928 06:49:40.423332 29742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.60:46753: Network error: Client connection negotiation failed: client connection to 127.7.48.60:46753: connect: Connection refused (error 111)
W20260928 06:49:40.439399 29824 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:49:40.439762 29824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:40.439824 29824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:40.439913 29824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 06:49:40.442812 29817 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } attempt: 2
I20260928 06:49:40.443661 29819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.443344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:46753 (local address 127.0.0.1:38102)
0928 06:49:40.443506 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.443572 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 06:49:40.444443 29824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:40.444505 29824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260928 06:49:40.444507 29819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.444280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:46753 (local address 127.0.0.1:38114)
0928 06:49:40.444388 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.444438 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 06:49:40.444540 29824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:40.444625 29824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:40.444677 29824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:49:40.444702 29817 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.60:46753: Network error: Client connection negotiation failed: client connection to 127.7.48.60:46753: connect: Connection refused (error 111)
I20260928 06:49:40.449987 29824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:35745,127.7.48.61:38121,127.7.48.60:46753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.60.keytab
--principal=kudu/127.7.48.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.60:46753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.48.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:40.451100 29824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:40.452226 29824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:40.460652 29832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:40.460637 29830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:40.460642 29829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:40.460994 29824 server_base.cc:1034] running on GCE node
I20260928 06:49:40.461653 29824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:40.462765 29824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:40.463951 29824 hybrid_clock.cc:648] HybridClock initialized: now 1790578180463919 us; error 55 us; skew 500 ppm
Sep 28 06:49:40 dist-test-slave-twl6 krb5kdc[29661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578180, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:40.466977 29824 init.cc:377] Logged in from keytab as kudu/127.7.48.60@KRBTEST.COM (short username kudu)
I20260928 06:49:40.468080 29824 webserver.cc:460] Webserver started at http://127.7.48.60:38543/ using document root <none> and password file <none>
I20260928 06:49:40.468704 29824 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:40.468804 29824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:40.469023 29824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:40.471374 29824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-2/data/instance:
uuid: "eaa38cb627a74e6cb47f049e96f5a435"
format_stamp: "Formatted at 2026-09-28 06:49:40 on dist-test-slave-twl6"
server_key: "bb2a23bf8581ee7840db2c790397fca1"
server_key_iv: "ddda91a23de4a193e2411fe982da08eb"
server_key_version: "encryptionkey@0"
I20260928 06:49:40.472038 29824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-2/wal/instance:
uuid: "eaa38cb627a74e6cb47f049e96f5a435"
format_stamp: "Formatted at 2026-09-28 06:49:40 on dist-test-slave-twl6"
server_key: "bb2a23bf8581ee7840db2c790397fca1"
server_key_iv: "ddda91a23de4a193e2411fe982da08eb"
server_key_version: "encryptionkey@0"
I20260928 06:49:40.475977 29824 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 06:49:40.478451 29839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:40.479776 29824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:49:40.479929 29824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-2/wal
uuid: "eaa38cb627a74e6cb47f049e96f5a435"
format_stamp: "Formatted at 2026-09-28 06:49:40 on dist-test-slave-twl6"
server_key: "bb2a23bf8581ee7840db2c790397fca1"
server_key_iv: "ddda91a23de4a193e2411fe982da08eb"
server_key_version: "encryptionkey@0"
I20260928 06:49:40.480062 29824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:40.498590 29824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:40.501930 29824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:40.502132 29824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:40.509938 29824 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.60:46753
I20260928 06:49:40.510221 29891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.60:46753 every 8 connection(s)
I20260928 06:49:40.510950 29824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-2/data/info.pb
I20260928 06:49:40.513782 29892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:40.517977 29892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.62" port: 35745 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:49:40.521752 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 29824
I20260928 06:49:40.522172 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577905526996-7360-0/minicluster-data/master-2/wal/instance
I20260928 06:49:40.522441 29742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } attempt: 3
I20260928 06:49:40.524874 7360 external_mini_cluster.cc:1428] Setting key 91000995afabc4526af1065329bdd68b
Sep 28 06:49:40 dist-test-slave-twl6 krb5kdc[29661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578180, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for kudu/127.7.48.60@KRBTEST.COM
Sep 28 06:49:40 dist-test-slave-twl6 krb5kdc[29661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578180, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.60@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
Sep 28 06:49:40 dist-test-slave-twl6 krb5kdc[29661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.60@KRBTEST.COM
I20260928 06:49:40.535904 29817 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } attempt: 3
I20260928 06:49:40.536942 29745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.522289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54874 (local address 127.7.48.62:35745)
0928 06:49:40.523275 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.523283 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.523306 (+ 23us) server_negotiation.cc:407] Connection header received
0928 06:49:40.524408 (+ 1102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.524413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.524466 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.524521 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.526337 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.527033 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.529050 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.529375 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.529449 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:40.532893 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:40.532896 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:40.532899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:40.532923 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:40.534550 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.535114 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.535118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.535119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.535195 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.535625 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.535629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.535630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.535999 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:40.536094 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:40.536657 (+ 563us) server_negotiation.cc:299] Negotiation successful
0928 06:49:40.536776 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 06:49:40.536969 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.523620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:46753 (local address 127.0.0.1:38128)
0928 06:49:40.523976 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.523993 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:40.524083 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:40.526217 (+ 2134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:40.526221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:40.526408 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:40.526497 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:40.526502 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.528046 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.528050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:40.528811 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:40.528821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.529586 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.529589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:40.529678 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:40.530244 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:40.530260 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:40.532005 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:40.534915 (+ 2910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:40.534930 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:40.534933 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:40.535195 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:40.535614 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:40.535618 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:40.535620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:40.535744 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:40.536570 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:40.536576 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:40.536658 (+ 82us) client_negotiation.cc:769] Sending connection context
0928 06:49:40.536734 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:49:40.536836 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225}
I20260928 06:49:40.538671 29894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.521212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:35745 (local address 127.0.0.1:54874)
0928 06:49:40.522919 (+ 1707us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.523045 (+ 126us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:40.524160 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:40.524950 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:40.524977 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:40.525498 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:40.526198 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:40.526214 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.527173 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.527176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:40.528939 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:40.528948 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.529631 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.529635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:40.530187 (+ 552us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:40.530779 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:40.530805 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:40.532736 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:40.534699 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:40.534708 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:40.534718 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:40.535001 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:40.535307 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:40.535310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:40.535311 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:40.535503 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:40.536159 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:40.536164 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:40.536405 (+ 241us) client_negotiation.cc:769] Sending connection context
0928 06:49:40.537565 (+ 1160us) client_negotiation.cc:240] Negotiation successful
0928 06:49:40.537725 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1454,"mutex_wait_us":507,"spinlock_wait_cycles":4608,"thread_start_us":167,"threads_started":1}
I20260928 06:49:40.538671 29895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.524258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38128 (local address 127.7.48.60:46753)
0928 06:49:40.525758 (+ 1500us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.525770 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.525793 (+ 23us) server_negotiation.cc:407] Connection header received
0928 06:49:40.525833 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.525838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.526014 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.526645 (+ 631us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.526762 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.527902 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.528953 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.529459 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.529916 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:40.532174 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:40.532181 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:40.532197 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:40.532240 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:40.534798 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.535321 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.535326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.535327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.535503 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.535903 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.535907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.535909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.536447 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:40.537181 (+ 734us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:40.537313 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 06:49:40.537608 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":1409,"spinlock_wait_cycles":4224,"thread_start_us":137,"threads_started":1}
I20260928 06:49:40.540526 29892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.61" port: 38121 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:49:40.542805 29896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.527508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38138 (local address 127.7.48.60:46753)
0928 06:49:40.527788 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.527798 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.528257 (+ 459us) server_negotiation.cc:407] Connection header received
0928 06:49:40.528485 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.528490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.528540 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.528603 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.529726 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.530681 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.532150 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.532568 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.532646 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:40.537785 (+ 5139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:40.537791 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:40.537796 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:40.537825 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:40.540423 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.541067 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.541072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.541077 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.541167 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.541493 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.541500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.541503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.541776 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:40.541939 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:40.542423 (+ 484us) server_negotiation.cc:299] Negotiation successful
0928 06:49:40.542559 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":209,"thread_start_us":84,"threads_started":1}
I20260928 06:49:40.543674 29742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630: Bootstrap starting.
Sep 28 06:49:40 dist-test-slave-twl6 krb5kdc[29661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578180, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.60@KRBTEST.COM for kudu/127.7.48.61@KRBTEST.COM
I20260928 06:49:40.546736 29742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:40.547657 29742 log.cc:824] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:40.550020 29742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630: No bootstrap required, opened a new log
I20260928 06:49:40.550595 29821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.541247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50952 (local address 127.7.48.61:38121)
0928 06:49:40.541359 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.541363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.541600 (+ 237us) server_negotiation.cc:407] Connection header received
0928 06:49:40.541771 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.541774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.541817 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.541863 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.542429 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.543143 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.544162 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.544484 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.544558 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:40.546851 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:40.546856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:40.546862 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:40.546889 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:40.548595 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.549066 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.549070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.549072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.549167 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.549488 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.549491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.549493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.549931 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:40.550005 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:40.550360 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 06:49:40.550462 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:49:40.551002 29894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.541001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:38121 (local address 127.0.0.1:50952)
0928 06:49:40.541531 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.541545 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:40.541667 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:40.541990 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:40.541992 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:40.542180 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:40.542270 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:40.542279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.543289 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.543292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:40.544016 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:40.544027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.544704 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.544707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:40.544786 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:40.545286 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:40.545301 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:40.546689 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:40.548733 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:40.548736 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:40.548738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:40.548956 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:40.549283 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:40.549287 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:40.549292 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:40.549389 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:40.550080 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:40.550083 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:40.550177 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 06:49:40.550757 (+ 580us) client_negotiation.cc:240] Negotiation successful
0928 06:49:40.550879 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458}
I20260928 06:49:40.551726 29892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:49:40 dist-test-slave-twl6 krb5kdc[29661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:49:40.555100 29742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c51594010848cbbc466d5cbadd3630" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 35745 } } peers { permanent_uuid: "676cf99ebbea4928b12aec4792fe64b6" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 38121 } } peers { permanent_uuid: "eaa38cb627a74e6cb47f049e96f5a435" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } }
I20260928 06:49:40.555464 29742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c51594010848cbbc466d5cbadd3630, State: Initialized, Role: FOLLOWER
Sep 28 06:49:40 dist-test-slave-twl6 krb5kdc[29661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578180, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.61@KRBTEST.COM for kudu/127.7.48.60@KRBTEST.COM
I20260928 06:49:40.555918 29742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c51594010848cbbc466d5cbadd3630" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 35745 } } peers { permanent_uuid: "676cf99ebbea4928b12aec4792fe64b6" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 38121 } } peers { permanent_uuid: "eaa38cb627a74e6cb47f049e96f5a435" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } }
I20260928 06:49:40.557483 29742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 06:49:40 dist-test-slave-twl6 krb5kdc[29661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578180, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.60@KRBTEST.COM for kudu/127.7.48.60@KRBTEST.COM
I20260928 06:49:40.561058 29745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.544798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54890 (local address 127.7.48.62:35745)
0928 06:49:40.544936 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.544942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.545203 (+ 261us) server_negotiation.cc:407] Connection header received
0928 06:49:40.545385 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.545390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.545445 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.545498 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.546478 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.547346 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.548666 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.549113 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.549200 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:40.554640 (+ 5440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:40.554646 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:40.554650 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:40.554682 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:40.557386 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.558377 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.558382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.558386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.558474 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.559214 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.559219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.559223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.559478 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:40.559658 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:40.560709 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0928 06:49:40.560835 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 06:49:40.564399 29911 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:49:40.564504 29911 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:49:40.566735 29819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.536437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:46753 (local address 127.0.0.1:38146)
0928 06:49:40.536785 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.536802 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:40.536906 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:40.549798 (+ 12892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:40.549804 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:40.550009 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:40.550131 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:40.550140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.552156 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.552159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:40.552893 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:40.552904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.553558 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.553562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:40.553658 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:40.554140 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:40.554154 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:40.556490 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:40.562135 (+ 5645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:40.562140 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:40.562144 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:40.562400 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:40.564411 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:40.564418 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:40.564423 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:40.564526 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:40.566076 (+ 1550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:40.566081 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:40.566211 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 06:49:40.566317 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 06:49:40.566494 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227}
I20260928 06:49:40.563038 29674 master_runner.cc:186] Error getting master registration for 127.7.48.61:38121: OK, instance_id { permanent_uuid: "676cf99ebbea4928b12aec4792fe64b6" instance_seqno: 1790578180282479 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:49:40.567422 29674 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:49:40.567699 29898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.537084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38146 (local address 127.7.48.60:46753)
0928 06:49:40.539428 (+ 2344us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.539439 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.539454 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:49:40.539492 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.539520 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.539574 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.550244 (+ 10670us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.550370 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.551201 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.553036 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.553383 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.554747 (+ 1364us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:40.557984 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:40.557991 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:40.557995 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:40.558027 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:40.561521 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.562849 (+ 1328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.562854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.562857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.562969 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.564657 (+ 1688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.564662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.564665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.564901 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:40.567197 (+ 2296us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:40.567288 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 06:49:40.567429 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1132,"thread_start_us":120,"threads_started":1}
I20260928 06:49:40.568054 29900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c51594010848cbbc466d5cbadd3630" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 35745 } } peers { permanent_uuid: "676cf99ebbea4928b12aec4792fe64b6" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 38121 } } peers { permanent_uuid: "eaa38cb627a74e6cb47f049e96f5a435" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } } }
I20260928 06:49:40.568169 29900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:49:40.568379 29896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.552511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.7.48.60:46753)
0928 06:49:40.552609 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.552613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.552639 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:49:40.554858 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.554861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.554894 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.555226 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.555705 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.556713 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.557628 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.557938 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.558011 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:40.561809 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:40.561814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:40.561817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:40.561840 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:40.563827 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.565064 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.565069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.565072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.565158 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.565452 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.565455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.565457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.565667 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:40.565744 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:40.568114 (+ 2370us) server_negotiation.cc:299] Negotiation successful
0928 06:49:40.568222 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:49:40.568838 29894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.552258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:46753 (local address 127.0.0.1:38154)
0928 06:49:40.552538 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:40.552555 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:40.552637 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:40.555326 (+ 2689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:40.555332 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:40.555499 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:40.555588 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:40.555596 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.556829 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.556831 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:40.557507 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:40.557519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.558150 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.558153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:40.558227 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:40.559092 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:40.559107 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:40.561696 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:40.563974 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:40.563980 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:40.563984 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:40.564240 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:40.565268 (+ 1028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:40.565272 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:40.565276 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:40.565358 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:40.565813 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:40.565817 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:40.565902 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 06:49:40.568489 (+ 2587us) client_negotiation.cc:240] Negotiation successful
0928 06:49:40.568631 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202}
I20260928 06:49:40.570989 29817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 676cf99ebbea4928b12aec4792fe64b6: Bootstrap starting.
Sep 28 06:49:40 dist-test-slave-twl6 krb5kdc[29661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.61@KRBTEST.COM
I20260928 06:49:40.574126 29817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 676cf99ebbea4928b12aec4792fe64b6: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:40.575053 29817 log.cc:824] T 00000000000000000000000000000000 P 676cf99ebbea4928b12aec4792fe64b6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:40.576047 29821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.565931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50956 (local address 127.7.48.61:38121)
0928 06:49:40.566044 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.566049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.566062 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:49:40.566095 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.566098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.566162 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.566244 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.567082 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.567732 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.568796 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.569060 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.569164 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:40.571990 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:40.571993 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:40.571996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:40.572021 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:40.573727 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.574247 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.574254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.574256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.574343 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.574654 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.574657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.574659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.574864 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:40.575043 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:40.575726 (+ 683us) server_negotiation.cc:299] Negotiation successful
0928 06:49:40.575842 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:49:40.578554 29892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaa38cb627a74e6cb47f049e96f5a435: Bootstrap starting.
I20260928 06:49:40.579689 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:49:40.580996 29716 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "66c51594010848cbbc466d5cbadd3630"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54890
I20260928 06:49:40.581264 29716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 06:49:40.581401 29716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:40.581591 29892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eaa38cb627a74e6cb47f049e96f5a435: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:40.582527 29892 log.cc:824] T 00000000000000000000000000000000 P eaa38cb627a74e6cb47f049e96f5a435: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:40.582856 29716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c51594010848cbbc466d5cbadd3630" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 35745 } } peers { permanent_uuid: "676cf99ebbea4928b12aec4792fe64b6" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 38121 } } peers { permanent_uuid: "eaa38cb627a74e6cb47f049e96f5a435" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } }
I20260928 06:49:40.583856 29716 leader_election.cc:288] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630 [CANDIDATE]: Term 1 election: Requested vote from peers 676cf99ebbea4928b12aec4792fe64b6 (127.7.48.61:38121), eaa38cb627a74e6cb47f049e96f5a435 (127.7.48.60:46753)
I20260928 06:49:40.586943 29817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 676cf99ebbea4928b12aec4792fe64b6: No bootstrap required, opened a new log
I20260928 06:49:40.590107 29892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaa38cb627a74e6cb47f049e96f5a435: No bootstrap required, opened a new log
I20260928 06:49:40.595163 29892 raft_consensus.cc:348] T 00000000000000000000000000000000 P eaa38cb627a74e6cb47f049e96f5a435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c51594010848cbbc466d5cbadd3630" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 35745 } } peers { permanent_uuid: "676cf99ebbea4928b12aec4792fe64b6" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 38121 } } peers { permanent_uuid: "eaa38cb627a74e6cb47f049e96f5a435" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } }
I20260928 06:49:40.595454 29892 raft_consensus.cc:729] T 00000000000000000000000000000000 P eaa38cb627a74e6cb47f049e96f5a435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa38cb627a74e6cb47f049e96f5a435, State: Initialized, Role: FOLLOWER
I20260928 06:49:40.595811 29892 consensus_queue.cc:260] T 00000000000000000000000000000000 P eaa38cb627a74e6cb47f049e96f5a435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c51594010848cbbc466d5cbadd3630" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 35745 } } peers { permanent_uuid: "676cf99ebbea4928b12aec4792fe64b6" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 38121 } } peers { permanent_uuid: "eaa38cb627a74e6cb47f049e96f5a435" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } }
I20260928 06:49:40.599656 29817 raft_consensus.cc:348] T 00000000000000000000000000000000 P 676cf99ebbea4928b12aec4792fe64b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c51594010848cbbc466d5cbadd3630" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 35745 } } peers { permanent_uuid: "676cf99ebbea4928b12aec4792fe64b6" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 38121 } } peers { permanent_uuid: "eaa38cb627a74e6cb47f049e96f5a435" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } }
I20260928 06:49:40.599859 29817 raft_consensus.cc:729] T 00000000000000000000000000000000 P 676cf99ebbea4928b12aec4792fe64b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 676cf99ebbea4928b12aec4792fe64b6, State: Initialized, Role: FOLLOWER
I20260928 06:49:40.600191 29817 consensus_queue.cc:260] T 00000000000000000000000000000000 P 676cf99ebbea4928b12aec4792fe64b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c51594010848cbbc466d5cbadd3630" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 35745 } } peers { permanent_uuid: "676cf99ebbea4928b12aec4792fe64b6" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 38121 } } peers { permanent_uuid: "eaa38cb627a74e6cb47f049e96f5a435" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } }
I20260928 06:49:40.600723 29923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 676cf99ebbea4928b12aec4792fe64b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c51594010848cbbc466d5cbadd3630" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 35745 } } peers { permanent_uuid: "676cf99ebbea4928b12aec4792fe64b6" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 38121 } } peers { permanent_uuid: "eaa38cb627a74e6cb47f049e96f5a435" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } } }
I20260928 06:49:40.601018 29923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 676cf99ebbea4928b12aec4792fe64b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:49:40.601591 29817 sys_catalog.cc:564] T 00000000000000000000000000000000 P 676cf99ebbea4928b12aec4792fe64b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:40.601678 29791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66c51594010848cbbc466d5cbadd3630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "676cf99ebbea4928b12aec4792fe64b6"
I20260928 06:49:40.601862 29791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 676cf99ebbea4928b12aec4792fe64b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:40.604184 29791 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 676cf99ebbea4928b12aec4792fe64b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66c51594010848cbbc466d5cbadd3630 in term 1.
I20260928 06:49:40.604735 29690 leader_election.cc:302] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66c51594010848cbbc466d5cbadd3630, 676cf99ebbea4928b12aec4792fe64b6; no voters:
I20260928 06:49:40.605127 29900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:40.605372 29900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630 [term 1 LEADER]: Becoming Leader. State: Replica: 66c51594010848cbbc466d5cbadd3630, State: Running, Role: LEADER
I20260928 06:49:40.605741 29900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c51594010848cbbc466d5cbadd3630" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 35745 } } peers { permanent_uuid: "676cf99ebbea4928b12aec4792fe64b6" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 38121 } } peers { permanent_uuid: "eaa38cb627a74e6cb47f049e96f5a435" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } }
I20260928 06:49:40.607573 29900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66c51594010848cbbc466d5cbadd3630. Latest consensus state: current_term: 1 leader_uuid: "66c51594010848cbbc466d5cbadd3630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c51594010848cbbc466d5cbadd3630" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 35745 } } peers { permanent_uuid: "676cf99ebbea4928b12aec4792fe64b6" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 38121 } } peers { permanent_uuid: "eaa38cb627a74e6cb47f049e96f5a435" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } } }
I20260928 06:49:40.607708 29900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:40.604517 29866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66c51594010848cbbc466d5cbadd3630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "eaa38cb627a74e6cb47f049e96f5a435"
I20260928 06:49:40.608276 29866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eaa38cb627a74e6cb47f049e96f5a435 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:40.609725 29866 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eaa38cb627a74e6cb47f049e96f5a435 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66c51594010848cbbc466d5cbadd3630 in term 1.
I20260928 06:49:40.604377 29892 sys_catalog.cc:564] T 00000000000000000000000000000000 P eaa38cb627a74e6cb47f049e96f5a435 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:40.618419 29745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.596335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54896 (local address 127.7.48.62:35745)
0928 06:49:40.596472 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.596477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.605705 (+ 9228us) server_negotiation.cc:407] Connection header received
0928 06:49:40.605827 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.605830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.605889 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.605981 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.606840 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.607563 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.608954 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.609253 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.609324 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:40.614777 (+ 5453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:40.614781 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:40.614783 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:40.614806 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:40.616577 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.617005 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.617008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.617010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.617105 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.617390 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.617393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.617395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.617658 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:40.617793 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:40.618142 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 06:49:40.618267 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:49:40.619387 29919 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaa38cb627a74e6cb47f049e96f5a435 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c51594010848cbbc466d5cbadd3630" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 35745 } } peers { permanent_uuid: "676cf99ebbea4928b12aec4792fe64b6" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 38121 } } peers { permanent_uuid: "eaa38cb627a74e6cb47f049e96f5a435" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } } }
I20260928 06:49:40.619500 29919 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaa38cb627a74e6cb47f049e96f5a435 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:49:40.620352 29937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:40.621374 29937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260928 06:49:40.626688 29934 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 676cf99ebbea4928b12aec4792fe64b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:49:40.626777 29934 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:49:40.628652 29898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.597033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38162 (local address 127.7.48.60:46753)
0928 06:49:40.597151 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.597155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.619161 (+ 22006us) server_negotiation.cc:407] Connection header received
0928 06:49:40.619332 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.619335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.619378 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.619427 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.620492 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.621187 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.622268 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.622586 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.622662 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:40.624361 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:40.624365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:40.624368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:40.624392 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:40.626239 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.626730 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.626734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.626737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.626831 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.627301 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.627305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.627308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.627559 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:40.627736 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:40.628357 (+ 621us) server_negotiation.cc:299] Negotiation successful
0928 06:49:40.628479 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 06:49:40.631150 29948 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eaa38cb627a74e6cb47f049e96f5a435: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:49:40.631232 29948 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:49:40.632849 29821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:40.596754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50972 (local address 127.7.48.61:38121)
0928 06:49:40.596885 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.596889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.613141 (+ 16252us) server_negotiation.cc:407] Connection header received
0928 06:49:40.613324 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.613327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.613373 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.613444 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.614633 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.615376 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.618271 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:40.618604 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:40.618690 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:40.626416 (+ 7726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:40.626422 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:40.626424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:40.626454 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:40.628217 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.631376 (+ 3159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.631379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.631381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.631473 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:40.631833 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:40.631837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:40.631839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:40.632086 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:40.632202 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:40.632571 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 06:49:40.632702 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:49:40.640893 29866 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eaa38cb627a74e6cb47f049e96f5a435 [term 1 FOLLOWER]: Refusing update from remote peer 66c51594010848cbbc466d5cbadd3630: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:49:40.640985 29791 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 676cf99ebbea4928b12aec4792fe64b6 [term 1 FOLLOWER]: Refusing update from remote peer 66c51594010848cbbc466d5cbadd3630: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:49:40.641553 29935 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaa38cb627a74e6cb47f049e96f5a435" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:49:40.641788 29900 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "676cf99ebbea4928b12aec4792fe64b6" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:49:40.645213 29919 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaa38cb627a74e6cb47f049e96f5a435 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66c51594010848cbbc466d5cbadd3630. Latest consensus state: current_term: 1 leader_uuid: "66c51594010848cbbc466d5cbadd3630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c51594010848cbbc466d5cbadd3630" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 35745 } } peers { permanent_uuid: "676cf99ebbea4928b12aec4792fe64b6" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 38121 } } peers { permanent_uuid: "eaa38cb627a74e6cb47f049e96f5a435" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } } }
I20260928 06:49:40.645326 29919 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaa38cb627a74e6cb47f049e96f5a435 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:49:40.645612 29923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 676cf99ebbea4928b12aec4792fe64b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66c51594010848cbbc466d5cbadd3630. Latest consensus state: current_term: 1 leader_uuid: "66c51594010848cbbc466d5cbadd3630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c51594010848cbbc466d5cbadd3630" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 35745 } } peers { permanent_uuid: "676cf99ebbea4928b12aec4792fe64b6" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 38121 } } peers { permanent_uuid: "eaa38cb627a74e6cb47f049e96f5a435" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } } }
I20260928 06:49:40.645764 29923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 676cf99ebbea4928b12aec4792fe64b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:49:40.651837 29900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66c51594010848cbbc466d5cbadd3630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c51594010848cbbc466d5cbadd3630" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 35745 } } peers { permanent_uuid: "676cf99ebbea4928b12aec4792fe64b6" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 38121 } } peers { permanent_uuid: "eaa38cb627a74e6cb47f049e96f5a435" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } } }
I20260928 06:49:40.651988 29900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:40.653020 29923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 676cf99ebbea4928b12aec4792fe64b6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66c51594010848cbbc466d5cbadd3630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c51594010848cbbc466d5cbadd3630" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 35745 } } peers { permanent_uuid: "676cf99ebbea4928b12aec4792fe64b6" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 38121 } } peers { permanent_uuid: "eaa38cb627a74e6cb47f049e96f5a435" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } } }
I20260928 06:49:40.653128 29923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 676cf99ebbea4928b12aec4792fe64b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:49:40.653556 29951 mvcc.cc:205] Tried to move back new op lower bound from 7334208227898413056 to 7334208227764883456. Current Snapshot: MvccSnapshot[applied={T|T < 7334208227898413056}]
I20260928 06:49:40.653935 29953 mvcc.cc:205] Tried to move back new op lower bound from 7334208227898413056 to 7334208227764883456. Current Snapshot: MvccSnapshot[applied={T|T < 7334208227898413056}]
I20260928 06:49:40.655260 29935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66c51594010848cbbc466d5cbadd3630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c51594010848cbbc466d5cbadd3630" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 35745 } } peers { permanent_uuid: "676cf99ebbea4928b12aec4792fe64b6" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 38121 } } peers { permanent_uuid: "eaa38cb627a74e6cb47f049e96f5a435" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } } }
I20260928 06:49:40.655395 29935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:40.657275 29919 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaa38cb627a74e6cb47f049e96f5a435 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66c51594010848cbbc466d5cbadd3630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c51594010848cbbc466d5cbadd3630" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 35745 } } peers { permanent_uuid: "676cf99ebbea4928b12aec4792fe64b6" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 38121 } } peers { permanent_uuid: "eaa38cb627a74e6cb47f049e96f5a435" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 46753 } } }
I20260928 06:49:40.657413 29919 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaa38cb627a74e6cb47f049e96f5a435 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:49:40.657733 29952 mvcc.cc:205] Tried to move back new op lower bound from 7334208227898413056 to 7334208227764883456. Current Snapshot: MvccSnapshot[applied={T|T < 7334208227898413056}]
I20260928 06:49:40.659967 29937 catalog_manager.cc:1347] Generated new cluster ID: c206b1fba9a44e0fb00e298bfd752208
I20260928 06:49:40.660038 29937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:40.680457 29937 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:40.681800 29937 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:40.704633 29937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66c51594010848cbbc466d5cbadd3630: Generated new TSK 0
I20260928 06:49:40.705590 29937 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:49:40.740554 29745 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:40.737954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54906 (local address 127.7.48.62:35745)
0928 06:49:40.738080 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.738085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.739356 (+ 1271us) server_negotiation.cc:407] Connection header received
0928 06:49:40.739555 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.739559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.739612 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.739679 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.740378 (+ 699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54906: BlockingRecv error: recv got EOF from 127.0.0.1:54906 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 06:49:40.741945 29898 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:40.738745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38178 (local address 127.7.48.60:46753)
0928 06:49:40.738864 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.738871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.740961 (+ 2090us) server_negotiation.cc:407] Connection header received
0928 06:49:40.741106 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.741109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.741172 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.741264 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.741810 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38178: BlockingRecv error: recv got EOF from 127.0.0.1:38178 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 06:49:40.743487 29821 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:40.742208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50974 (local address 127.7.48.61:38121)
0928 06:49:40.742323 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.742327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.742632 (+ 305us) server_negotiation.cc:407] Connection header received
0928 06:49:40.742796 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.742800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.742846 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.742895 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.743370 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50974: BlockingRecv error: recv got EOF from 127.0.0.1:50974 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 06:49:40.748039 29821 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:40.746469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50980 (local address 127.7.48.61:38121)
0928 06:49:40.746565 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.746569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.747080 (+ 511us) server_negotiation.cc:407] Connection header received
0928 06:49:40.747290 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.747293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.747364 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.747414 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.747935 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50980: BlockingRecv error: recv got EOF from 127.0.0.1:50980 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 06:49:40.753152 29821 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:40.751653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50990 (local address 127.7.48.61:38121)
0928 06:49:40.751753 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.751757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.752256 (+ 499us) server_negotiation.cc:407] Connection header received
0928 06:49:40.752440 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.752443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.752491 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.752552 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.753053 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50990: BlockingRecv error: recv got EOF from 127.0.0.1:50990 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 06:49:40.765352 29821 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:40.760298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51000 (local address 127.7.48.61:38121)
0928 06:49:40.760413 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.760417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.764256 (+ 3839us) server_negotiation.cc:407] Connection header received
0928 06:49:40.764494 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.764498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.764557 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.764616 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.765236 (+ 620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51000: BlockingRecv error: recv got EOF from 127.0.0.1:51000 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 06:49:40.773847 29821 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:40.772057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51010 (local address 127.7.48.61:38121)
0928 06:49:40.772198 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.772202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.772794 (+ 592us) server_negotiation.cc:407] Connection header received
0928 06:49:40.773023 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.773026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.773081 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.773137 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.773726 (+ 589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51010: BlockingRecv error: recv got EOF from 127.0.0.1:51010 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 06:49:40.786600 29821 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:40.784758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51024 (local address 127.7.48.61:38121)
0928 06:49:40.784888 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.784893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.785531 (+ 638us) server_negotiation.cc:407] Connection header received
0928 06:49:40.785756 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.785760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.785815 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.785871 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.786481 (+ 610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51024: BlockingRecv error: recv got EOF from 127.0.0.1:51024 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 06:49:40.807461 29821 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:40.805475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51034 (local address 127.7.48.61:38121)
0928 06:49:40.805605 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.805610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.806381 (+ 771us) server_negotiation.cc:407] Connection header received
0928 06:49:40.806606 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.806610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.806666 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.806725 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.807327 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51034: BlockingRecv error: recv got EOF from 127.0.0.1:51034 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 06:49:40.843303 29821 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:40.842242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51048 (local address 127.7.48.61:38121)
0928 06:49:40.842371 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.842375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.842541 (+ 166us) server_negotiation.cc:407] Connection header received
0928 06:49:40.842645 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.842647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.842691 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.842768 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.843174 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51048: BlockingRecv error: recv got EOF from 127.0.0.1:51048 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 06:49:40.911124 29821 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:40.909717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51056 (local address 127.7.48.61:38121)
0928 06:49:40.909919 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:40.909925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:40.910086 (+ 161us) server_negotiation.cc:407] Connection header received
0928 06:49:40.910276 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:40.910279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:40.910326 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:40.910418 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:40.910998 (+ 580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51056: BlockingRecv error: recv got EOF from 127.0.0.1:51056 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260928 06:49:41.042441 29821 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:41.041214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51062 (local address 127.7.48.61:38121)
0928 06:49:41.041385 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:41.041392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:41.041541 (+ 149us) server_negotiation.cc:407] Connection header received
0928 06:49:41.041668 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:41.041671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:41.041719 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:41.041825 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:41.042315 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51062: BlockingRecv error: recv got EOF from 127.0.0.1:51062 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 06:49:41.301777 29821 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:41.300530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51074 (local address 127.7.48.61:38121)
0928 06:49:41.300680 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:41.300684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:41.300817 (+ 133us) server_negotiation.cc:407] Connection header received
0928 06:49:41.300997 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:41.301000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:41.301052 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:41.301150 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:41.301643 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51074: BlockingRecv error: recv got EOF from 127.0.0.1:51074 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 06:49:41.628120 29934 catalog_manager.cc:1259] Loaded cluster ID: c206b1fba9a44e0fb00e298bfd752208
I20260928 06:49:41.628203 29934 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 676cf99ebbea4928b12aec4792fe64b6: loading cluster ID for follower catalog manager: success
I20260928 06:49:41.629745 29934 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 676cf99ebbea4928b12aec4792fe64b6: acquiring CA information for follower catalog manager: success
I20260928 06:49:41.630788 29934 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 676cf99ebbea4928b12aec4792fe64b6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:49:41.632342 29948 catalog_manager.cc:1259] Loaded cluster ID: c206b1fba9a44e0fb00e298bfd752208
I20260928 06:49:41.632407 29948 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eaa38cb627a74e6cb47f049e96f5a435: loading cluster ID for follower catalog manager: success
I20260928 06:49:41.633921 29948 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eaa38cb627a74e6cb47f049e96f5a435: acquiring CA information for follower catalog manager: success
I20260928 06:49:41.634953 29948 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eaa38cb627a74e6cb47f049e96f5a435: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:49:41.819607 30023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:41.815863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51090 (local address 127.7.48.61:38121)
0928 06:49:41.816153 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:41.816156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:41.816253 (+ 97us) server_negotiation.cc:407] Connection header received
0928 06:49:41.816359 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:41.816361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:41.816487 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:41.816595 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:49:41.816827 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:41.817355 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:41.818162 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:41.818375 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:41.818448 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:41.818725 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:49:41.819176 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:49:41.819268 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:41.819356 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 06:49:41.819406 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":93,"threads_started":1}
I20260928 06:49:41.826566 30029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:41.822895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38184 (local address 127.7.48.60:46753)
0928 06:49:41.823145 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:41.823149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:41.823161 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:49:41.823241 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:41.823243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:41.823358 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:41.823470 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:49:41.823661 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:41.824240 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:41.825022 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:41.825250 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:41.825349 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:41.825638 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:49:41.826155 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:49:41.826253 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:41.826322 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 06:49:41.826371 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":88,"threads_started":1}
I20260928 06:49:41.834039 30023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:41.829815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51094 (local address 127.7.48.61:38121)
0928 06:49:41.829950 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:41.829955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:41.830138 (+ 183us) server_negotiation.cc:407] Connection header received
0928 06:49:41.830497 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:41.830500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:41.830560 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:41.830643 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:49:41.830929 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:41.831569 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:41.832708 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:41.833045 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:41.833161 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:41.833470 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:49:41.833551 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:49:41.833646 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:41.833783 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 06:49:41.833848 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:49:41.834311 30029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:41.830253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38192 (local address 127.7.48.60:46753)
0928 06:49:41.830374 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:41.830378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:41.830638 (+ 260us) server_negotiation.cc:407] Connection header received
0928 06:49:41.830787 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:41.830791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:41.830848 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:41.830920 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:49:41.831212 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:41.832006 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:41.833019 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:41.833352 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:41.833461 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:41.833843 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:49:41.833915 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:49:41.833961 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:41.834139 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 06:49:41.834193 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 06:49:41.834761 30035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:41.829839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54910 (local address 127.7.48.62:35745)
0928 06:49:41.830383 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:41.830388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:41.830403 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:49:41.830614 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:41.830621 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:41.830802 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:41.830936 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:49:41.831281 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:41.832096 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:41.833148 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:41.833523 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:41.833626 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:41.833988 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:49:41.834365 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:49:41.834473 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:41.834586 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 06:49:41.834653 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":60,"threads_started":1}
I20260928 06:49:41.838469 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 29674
I20260928 06:49:41.845572 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 29749
I20260928 06:49:41.851717 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 29824
2026-09-28T06:49:41Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3096 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:49:41 dist-test-slave-twl6 krb5kdc[30040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:49:41 dist-test-slave-twl6 krb5kdc[30040](info): set up 2 sockets
Sep 28 06:49:41 dist-test-slave-twl6 krb5kdc[30040](info): commencing operation
krb5kdc: starting...
W20260928 06:49:42.872646 7360 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/krb5kdc/test-user.keytab.
Sep 28 06:49:42 dist-test-slave-twl6 krb5kdc[30040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:49:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:49:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.48.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
Entry for principal kudu/127.7.48.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab.
I20260928 06:49:42.995246 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:33303
--webserver_interface=127.7.48.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:33303,127.7.48.61:44341,127.7.48.60:39611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:49:43.110203 30053 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:49:43.110589 30053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:43.110667 30053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:43.110741 30053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:43.115001 30053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:43.115115 30053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:43.115161 30053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:43.115200 30053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:43.115236 30053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:43.120896 30053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:33303,127.7.48.61:44341,127.7.48.60:39611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.62.keytab
--principal=kudu/127.7.48.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.62:33303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.48.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:43.122083 30053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:43.123102 30053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:43.132149 30061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:43.132115 30058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:43.132268 30053 server_base.cc:1034] running on GCE node
W20260928 06:49:43.132138 30059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:43.133122 30053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:43.134318 30053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:43.135511 30053 hybrid_clock.cc:648] HybridClock initialized: now 1790578183135481 us; error 48 us; skew 500 ppm
Sep 28 06:49:43 dist-test-slave-twl6 krb5kdc[30040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578183, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:43.138371 30053 init.cc:377] Logged in from keytab as kudu/127.7.48.62@KRBTEST.COM (short username kudu)
I20260928 06:49:43.139247 30053 webserver.cc:460] Webserver started at http://127.7.48.62:33905/ using document root <none> and password file <none>
I20260928 06:49:43.139776 30053 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:43.139834 30053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:43.140079 30053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:43.142105 30053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-0/data/instance:
uuid: "5b71d36de6b8455290a0b9e0e2bf14e9"
format_stamp: "Formatted at 2026-09-28 06:49:43 on dist-test-slave-twl6"
server_key: "be4e5d3690720bbf22c8d5d44e27caaa"
server_key_iv: "3a0b85b897a2eb771aad8a181b180f79"
server_key_version: "encryptionkey@0"
I20260928 06:49:43.142616 30053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-0/wal/instance:
uuid: "5b71d36de6b8455290a0b9e0e2bf14e9"
format_stamp: "Formatted at 2026-09-28 06:49:43 on dist-test-slave-twl6"
server_key: "be4e5d3690720bbf22c8d5d44e27caaa"
server_key_iv: "3a0b85b897a2eb771aad8a181b180f79"
server_key_version: "encryptionkey@0"
I20260928 06:49:43.146469 30053 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:49:43.149001 30068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:43.150283 30053 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:49:43.150388 30053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-0/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-0/wal
uuid: "5b71d36de6b8455290a0b9e0e2bf14e9"
format_stamp: "Formatted at 2026-09-28 06:49:43 on dist-test-slave-twl6"
server_key: "be4e5d3690720bbf22c8d5d44e27caaa"
server_key_iv: "3a0b85b897a2eb771aad8a181b180f79"
server_key_version: "encryptionkey@0"
I20260928 06:49:43.150480 30053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:43.169524 30053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:43.172614 30053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:43.172806 30053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:43.180122 30053 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.62:33303
I20260928 06:49:43.180475 30120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.62:33303 every 8 connection(s)
I20260928 06:49:43.181254 30053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-0/data/info.pb
I20260928 06:49:43.182317 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 30053
I20260928 06:49:43.182437 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-0/wal/instance
I20260928 06:49:43.182756 7360 external_mini_cluster.cc:1428] Setting key 9464771cba58219508e2fffe640de080
I20260928 06:49:43.184980 30121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:43.189142 30121 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33303 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.48.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.61@KRBTEST.COM" created.
Sep 28 06:49:43 dist-test-slave-twl6 krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578183, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:49:43.206207 30125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.191799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47460 (local address 127.7.48.62:33303)
0928 06:49:43.192426 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:43.192442 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:43.192655 (+ 213us) server_negotiation.cc:407] Connection header received
0928 06:49:43.194187 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:43.194192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:43.194411 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:43.194606 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:43.195518 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.196625 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.197685 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.198080 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.198255 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:43.200844 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:43.200850 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:43.200861 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:43.200882 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:43.202596 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.203117 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.203129 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.203130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.203233 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.203625 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.203631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.203632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.204178 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:43.204321 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:43.204720 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 06:49:43.204999 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"spinlock_wait_cycles":38656,"thread_start_us":196,"threads_started":1}
I20260928 06:49:43.206207 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.191799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33303 (local address 127.0.0.1:47460)
0928 06:49:43.192464 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.192516 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:43.193820 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:43.194652 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:43.194660 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:43.195074 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:43.195349 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:43.195363 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.196744 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.196747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:43.197555 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:43.197565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.198249 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.198253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:43.198365 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:43.199037 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:43.199054 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:43.200673 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:43.202714 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:43.202725 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:43.202737 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:43.203000 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:43.203330 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:43.203335 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:43.203337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:43.203450 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:43.204299 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:43.204305 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:43.204457 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 06:49:43.204647 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 06:49:43.205107 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":145,"thread_start_us":169,"threads_started":1}
I20260928 06:49:43.209064 30121 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.61" port: 44341 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:49:43.209918 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.209584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:44341 (local address 127.0.0.1:45524)
0928 06:49:43.209767 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.209846 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.61:44341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260928 06:49:43.210134 30070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.48.61:44341: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.61.keytab.
Entry for principal kudu/127.7.48.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.61.keytab.
I20260928 06:49:43.211138 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.210853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:44341 (local address 127.0.0.1:45538)
0928 06:49:43.210980 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.211053 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.61:44341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 06:49:43.211308 30121 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.61:44341: Network error: Client connection negotiation failed: client connection to 127.7.48.61:44341: connect: Connection refused (error 111)
I20260928 06:49:43.211357 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.61:44341
--webserver_interface=127.7.48.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.61.keytab
--principal=kudu/127.7.48.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:33303,127.7.48.61:44341,127.7.48.60:39611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:49:43.260613 30121 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.61" port: 44341 } attempt: 1
I20260928 06:49:43.261569 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.261212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:44341 (local address 127.0.0.1:45550)
0928 06:49:43.261390 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.261468 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.61:44341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 06:49:43.262296 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.262075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:44341 (local address 127.0.0.1:45554)
0928 06:49:43.262180 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.262232 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.61:44341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 06:49:43.262428 30121 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.61:44341: Network error: Client connection negotiation failed: client connection to 127.7.48.61:44341: connect: Connection refused (error 111)
W20260928 06:49:43.327822 30128 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:49:43.328155 30128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:43.328179 30128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:43.328294 30128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 06:49:43.330590 30121 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.61" port: 44341 } attempt: 2
I20260928 06:49:43.331458 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.331123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:44341 (local address 127.0.0.1:45558)
0928 06:49:43.331292 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.331366 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.61:44341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 06:49:43.332249 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.332024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:44341 (local address 127.0.0.1:45566)
0928 06:49:43.332131 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.332184 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.61:44341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 06:49:43.332384 30121 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.61:44341: Network error: Client connection negotiation failed: client connection to 127.7.48.61:44341: connect: Connection refused (error 111)
W20260928 06:49:43.333176 30128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:43.333230 30128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:43.333274 30128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:43.333312 30128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:43.333346 30128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:43.338989 30128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:33303,127.7.48.61:44341,127.7.48.60:39611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.61.keytab
--principal=kudu/127.7.48.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.61:44341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.48.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:43.340294 30128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:43.341396 30128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:43.350235 30133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:43.350229 30134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:43.350240 30136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:43.350333 30128 server_base.cc:1034] running on GCE node
I20260928 06:49:43.351023 30128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:43.352217 30128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:43.353397 30128 hybrid_clock.cc:648] HybridClock initialized: now 1790578183353385 us; error 28 us; skew 500 ppm
Sep 28 06:49:43 dist-test-slave-twl6 krb5kdc[30040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578183, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:43.356312 30128 init.cc:377] Logged in from keytab as kudu/127.7.48.61@KRBTEST.COM (short username kudu)
I20260928 06:49:43.357267 30128 webserver.cc:460] Webserver started at http://127.7.48.61:33757/ using document root <none> and password file <none>
I20260928 06:49:43.357851 30128 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:43.357940 30128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:43.358188 30128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:43.360376 30128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-1/data/instance:
uuid: "695ca52f54024cbf9d32b2bd54fccaf9"
format_stamp: "Formatted at 2026-09-28 06:49:43 on dist-test-slave-twl6"
server_key: "b729fe398f6faff48410778289e015ff"
server_key_iv: "74340ee5a970ff9e855743ca49c2622c"
server_key_version: "encryptionkey@0"
I20260928 06:49:43.360997 30128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-1/wal/instance:
uuid: "695ca52f54024cbf9d32b2bd54fccaf9"
format_stamp: "Formatted at 2026-09-28 06:49:43 on dist-test-slave-twl6"
server_key: "b729fe398f6faff48410778289e015ff"
server_key_iv: "74340ee5a970ff9e855743ca49c2622c"
server_key_version: "encryptionkey@0"
I20260928 06:49:43.365005 30128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 06:49:43.367563 30143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:43.368875 30128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 06:49:43.369020 30128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-1/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-1/wal
uuid: "695ca52f54024cbf9d32b2bd54fccaf9"
format_stamp: "Formatted at 2026-09-28 06:49:43 on dist-test-slave-twl6"
server_key: "b729fe398f6faff48410778289e015ff"
server_key_iv: "74340ee5a970ff9e855743ca49c2622c"
server_key_version: "encryptionkey@0"
I20260928 06:49:43.369148 30128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:43.387157 30128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:43.390282 30128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:43.390504 30128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:43.397954 30128 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.61:44341
I20260928 06:49:43.398262 30195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.61:44341 every 8 connection(s)
I20260928 06:49:43.399036 30128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-1/data/info.pb
I20260928 06:49:43.401868 30196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:43.405674 30196 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33303 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:49:43.408488 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 30128
I20260928 06:49:43.408636 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-1/wal/instance
I20260928 06:49:43.408953 7360 external_mini_cluster.cc:1428] Setting key 9d03d413a54585deae3a5da8a3ca3fd5
Sep 28 06:49:43 dist-test-slave-twl6 krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578183, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.61@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.48.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.48.60@KRBTEST.COM" created.
I20260928 06:49:43.420137 30125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.407579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47472 (local address 127.7.48.62:33303)
0928 06:49:43.407750 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:43.407755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:43.408757 (+ 1002us) server_negotiation.cc:407] Connection header received
0928 06:49:43.410135 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:43.410138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:43.410193 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:43.410272 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:43.411610 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.412214 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.413477 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.413724 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.413841 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:43.416399 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:43.416402 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:43.416405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:43.416430 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:43.417861 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.418404 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.418412 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.418414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.418491 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.418850 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.418853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.418855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.419044 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:43.419159 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:43.419829 (+ 670us) server_negotiation.cc:299] Negotiation successful
0928 06:49:43.420004 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 06:49:43.421382 30198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.408010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33303 (local address 127.0.0.1:47472)
0928 06:49:43.408563 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.408636 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:43.409893 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:43.410481 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:43.410494 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:43.410953 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:43.411446 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:43.411460 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.412351 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.412359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:43.413337 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:43.413346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.413888 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.413892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:43.414061 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:43.414708 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:43.414725 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:43.416219 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:43.417993 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:43.418000 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:43.418015 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:43.418288 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:43.418595 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:43.418598 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:43.418600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:43.418747 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:43.419164 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:43.419173 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:43.419527 (+ 354us) client_negotiation.cc:769] Sending connection context
0928 06:49:43.419762 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 06:49:43.420194 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":133,"threads_started":1}
I20260928 06:49:43.422819 30196 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.61" port: 44341 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:49:43.423527 30121 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.61" port: 44341 } attempt: 3
Sep 28 06:49:43 dist-test-slave-twl6 krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578183, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for kudu/127.7.48.61@KRBTEST.COM
Sep 28 06:49:43 dist-test-slave-twl6 krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578183, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.61@KRBTEST.COM for kudu/127.7.48.61@KRBTEST.COM
I20260928 06:49:43.436216 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.423929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:44341 (local address 127.0.0.1:45594)
0928 06:49:43.424133 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.424151 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:43.424236 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:43.424693 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:43.424696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:43.424890 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:43.424985 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:43.424991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.426026 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.426030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:43.427350 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:43.427363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.428072 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.428076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:43.428199 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:43.428732 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:43.428768 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:43.430928 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:43.434412 (+ 3484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:43.434418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:43.434422 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:43.434722 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:43.435060 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:43.435064 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:43.435066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:43.435161 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:43.435573 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:43.435576 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:43.435698 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 06:49:43.435832 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 06:49:43.436016 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 06:49:43.436259 30198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.423420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:44341 (local address 127.0.0.1:45582)
0928 06:49:43.423648 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.423662 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:43.423747 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:43.424731 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:43.424737 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:43.424944 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:43.425107 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:43.425117 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.426353 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.426357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:43.427248 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:43.427259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.428103 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.428110 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:43.428228 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:43.428966 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:43.428978 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:43.431492 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:43.434327 (+ 2835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:43.434330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:43.434332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:43.434594 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:43.434915 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:43.434918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:43.434920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:43.435003 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:43.435569 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:43.435572 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:43.435657 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 06:49:43.435823 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 06:49:43.436051 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260928 06:49:43.436275 30201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.423533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45582 (local address 127.7.48.61:44341)
0928 06:49:43.424243 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:43.424259 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:43.424287 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:49:43.424340 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:43.424349 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:43.424579 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:43.425229 (+ 650us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:43.425330 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.426226 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.427392 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.427960 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.428467 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:43.431705 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:43.431711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:43.431718 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:43.431763 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:43.434191 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.434706 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.434714 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.434715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.434808 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.435093 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.435097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.435099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.435452 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:43.435596 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:43.435902 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 06:49:43.436112 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":459,"spinlock_wait_cycles":3456,"thread_start_us":123,"threads_started":1}
I20260928 06:49:43.436800 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.423962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45594 (local address 127.7.48.61:44341)
0928 06:49:43.424289 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:43.424299 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:43.424313 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:49:43.424354 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:43.424356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:43.424549 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:43.424700 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:43.425101 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.425898 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.427485 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.427953 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.428068 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:43.431092 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:43.431103 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:43.431120 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:43.431165 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:43.434243 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.434877 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.434879 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.434880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.434955 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.435269 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.435273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.435275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.435469 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:43.435596 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:43.435923 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 06:49:43.436208 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":186,"server-negotiator.queue_time_us":256,"spinlock_wait_cycles":9216,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.48.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.60.keytab.
Entry for principal kudu/127.7.48.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.60.keytab.
I20260928 06:49:43.438674 30121 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:49:43.438759 30196 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:49:43.439201 7360 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskipM9PC/build/debug/bin/kudu
/tmp/dist-test-taskipM9PC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.60:39611
--webserver_interface=127.7.48.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.60.keytab
--principal=kudu/127.7.48.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.48.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.48.62:33303,127.7.48.61:44341,127.7.48.60:39611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:49:43.439491 30198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.439195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:39611 (local address 127.0.0.1:54194)
0928 06:49:43.439368 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.439436 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 06:49:43.439522 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.439195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:39611 (local address 127.0.0.1:54186)
0928 06:49:43.439356 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.439430 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 06:49:43.439697 30147 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.48.60:39611: connect: Connection refused (error 111)
I20260928 06:49:43.440378 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.440044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:39611 (local address 127.0.0.1:54200)
0928 06:49:43.440247 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.440300 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260928 06:49:43.440503 30121 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.60:39611: Network error: Client connection negotiation failed: client connection to 127.7.48.60:39611: connect: Connection refused (error 111)
I20260928 06:49:43.440717 30198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.440405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:39611 (local address 127.0.0.1:54216)
0928 06:49:43.440583 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.440643 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260928 06:49:43.441076 30196 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.60:39611: Network error: Client connection negotiation failed: client connection to 127.7.48.60:39611: connect: Connection refused (error 111)
I20260928 06:49:43.471732 30121 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } attempt: 1
I20260928 06:49:43.472683 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.472369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:39611 (local address 127.0.0.1:54218)
0928 06:49:43.472524 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.472600 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 06:49:43.473696 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.473392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:39611 (local address 127.0.0.1:54222)
0928 06:49:43.473553 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.473619 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 06:49:43.473856 30121 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.60:39611: Network error: Client connection negotiation failed: client connection to 127.7.48.60:39611: connect: Connection refused (error 111)
I20260928 06:49:43.490401 30196 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } attempt: 1
I20260928 06:49:43.491312 30198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.491003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:39611 (local address 127.0.0.1:54232)
0928 06:49:43.491164 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.491234 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 06:49:43.492154 30198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.491930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:39611 (local address 127.0.0.1:54244)
0928 06:49:43.492036 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.492096 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 06:49:43.492332 30196 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.60:39611: Network error: Client connection negotiation failed: client connection to 127.7.48.60:39611: connect: Connection refused (error 111)
I20260928 06:49:43.549033 30121 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } attempt: 2
I20260928 06:49:43.549897 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.549589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:39611 (local address 127.0.0.1:54258)
0928 06:49:43.549751 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.549820 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 06:49:43.550729 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.550518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:39611 (local address 127.0.0.1:54270)
0928 06:49:43.550628 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.550674 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 06:49:43.550943 30121 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.60:39611: Network error: Client connection negotiation failed: client connection to 127.7.48.60:39611: connect: Connection refused (error 111)
W20260928 06:49:43.555380 30204 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:49:43.555713 30204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:49:43.555761 30204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:49:43.555847 30204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:49:43.560055 30204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:49:43.560115 30204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:49:43.560158 30204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:49:43.560194 30204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:49:43.560231 30204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:49:43.560482 30196 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } attempt: 2
I20260928 06:49:43.561357 30198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.561061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:39611 (local address 127.0.0.1:54280)
0928 06:49:43.561220 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.561282 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 06:49:43.562132 30198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.561935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:39611 (local address 127.0.0.1:54284)
0928 06:49:43.562032 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.562075 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.48.60:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 06:49:43.562322 30196 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.48.60:39611: Network error: Client connection negotiation failed: client connection to 127.7.48.60:39611: connect: Connection refused (error 111)
I20260928 06:49:43.566452 30204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.48.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.48.62:33303,127.7.48.61:44341,127.7.48.60:39611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/krb5kdc/kudu_127.7.48.60.keytab
--principal=kudu/127.7.48.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.48.60:39611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.48.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:49:43.567610 30204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:49:43.568737 30204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:49:43.577350 30212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:43.577350 30209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:49:43.577356 30210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:49:43.577598 30204 server_base.cc:1034] running on GCE node
I20260928 06:49:43.578354 30204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:49:43.579504 30204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:49:43.580714 30204 hybrid_clock.cc:648] HybridClock initialized: now 1790578183580669 us; error 34 us; skew 500 ppm
Sep 28 06:49:43 dist-test-slave-twl6 krb5kdc[30040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578183, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:49:43.583489 30204 init.cc:377] Logged in from keytab as kudu/127.7.48.60@KRBTEST.COM (short username kudu)
I20260928 06:49:43.584517 30204 webserver.cc:460] Webserver started at http://127.7.48.60:34667/ using document root <none> and password file <none>
I20260928 06:49:43.585127 30204 fs_manager.cc:359] Metadata directory not provided
I20260928 06:49:43.585227 30204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:49:43.585458 30204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:49:43.587548 30204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-2/data/instance:
uuid: "d3a21b882a294a98a63adcb4e2b3368b"
format_stamp: "Formatted at 2026-09-28 06:49:43 on dist-test-slave-twl6"
server_key: "caf31999c0856fd8e39e96b15e39e14e"
server_key_iv: "ca8cf246c3c7845ff47ec51fbcef6118"
server_key_version: "encryptionkey@0"
I20260928 06:49:43.588193 30204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-2/wal/instance:
uuid: "d3a21b882a294a98a63adcb4e2b3368b"
format_stamp: "Formatted at 2026-09-28 06:49:43 on dist-test-slave-twl6"
server_key: "caf31999c0856fd8e39e96b15e39e14e"
server_key_iv: "ca8cf246c3c7845ff47ec51fbcef6118"
server_key_version: "encryptionkey@0"
I20260928 06:49:43.592146 30204 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 06:49:43.594547 30219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:49:43.595809 30204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:49:43.595979 30204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-2/data,/tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-2/wal
uuid: "d3a21b882a294a98a63adcb4e2b3368b"
format_stamp: "Formatted at 2026-09-28 06:49:43 on dist-test-slave-twl6"
server_key: "caf31999c0856fd8e39e96b15e39e14e"
server_key_iv: "ca8cf246c3c7845ff47ec51fbcef6118"
server_key_version: "encryptionkey@0"
I20260928 06:49:43.596107 30204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:49:43.609205 30204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:49:43.612116 30204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:49:43.612322 30204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:49:43.619390 30204 rpc_server.cc:307] RPC server started. Bound to: 127.7.48.60:39611
I20260928 06:49:43.619688 30271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.48.60:39611 every 8 connection(s)
I20260928 06:49:43.620448 30204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-2/data/info.pb
I20260928 06:49:43.623229 30272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:49:43.626284 7360 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskipM9PC/build/debug/bin/kudu as pid 30204
I20260928 06:49:43.626415 7360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskipM9PC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577905526996-7360-0/minicluster-data/master-2/wal/instance
I20260928 06:49:43.626729 7360 external_mini_cluster.cc:1428] Setting key e0d933b3eaaf45f2c9b4bc9b7413cb64
I20260928 06:49:43.627452 30272 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33303 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:49:43 dist-test-slave-twl6 krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578183, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.60@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
Sep 28 06:49:43 dist-test-slave-twl6 krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.60@KRBTEST.COM
I20260928 06:49:43.639343 30125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.629403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47486 (local address 127.7.48.62:33303)
0928 06:49:43.629535 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:43.629540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:43.629958 (+ 418us) server_negotiation.cc:407] Connection header received
0928 06:49:43.630093 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:43.630095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:43.630139 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:43.630240 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:43.631048 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.631555 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.632607 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.632822 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.632934 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:43.635530 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:43.635536 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:43.635541 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:43.635567 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:43.637040 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.637610 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.637613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.637615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.637698 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.638054 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.638057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.638058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.638291 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:43.638402 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:43.639018 (+ 616us) server_negotiation.cc:299] Negotiation successful
0928 06:49:43.639162 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 06:49:43.640225 30276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.629243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.62:33303 (local address 127.0.0.1:47486)
0928 06:49:43.629655 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.629707 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:43.629888 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:43.630297 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:43.630302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:43.630654 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:43.630903 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:43.630916 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.631671 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.631675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:43.632475 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:43.632484 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.632970 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.632974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:43.633149 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:43.633766 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:43.633791 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:43.635384 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:43.637153 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:43.637164 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:43.637174 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:43.637499 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:43.637793 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:43.637796 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:43.637798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:43.637961 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:43.638407 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:43.638412 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:43.638747 (+ 335us) client_negotiation.cc:769] Sending connection context
0928 06:49:43.638918 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 06:49:43.639237 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":157,"threads_started":1}
I20260928 06:49:43.640226 30275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.628175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54300 (local address 127.7.48.60:39611)
0928 06:49:43.628613 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:43.628629 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:43.628667 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:49:43.629408 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:43.629453 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:43.629917 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:43.630148 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:43.631024 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.631851 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.632755 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.633111 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.633226 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:43.636137 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:43.636146 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:43.636166 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:43.636188 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:43.637886 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.638332 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.638340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.638342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.638432 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.638716 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.638718 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.638719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.638979 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:43.639177 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:43.639695 (+ 518us) server_negotiation.cc:299] Negotiation successful
0928 06:49:43.639836 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":142,"threads_started":1}
I20260928 06:49:43.641512 30272 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.61" port: 44341 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:49:43 dist-test-slave-twl6 krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578183, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.60@KRBTEST.COM for kudu/127.7.48.61@KRBTEST.COM
Sep 28 06:49:43 dist-test-slave-twl6 krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.62@KRBTEST.COM
I20260928 06:49:43.650115 30121 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } attempt: 3
I20260928 06:49:43.650398 30276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.642006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.61:44341 (local address 127.0.0.1:45608)
0928 06:49:43.642126 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.642144 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:43.642213 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:43.642470 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:43.642472 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:43.642609 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:43.642764 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:43.642773 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.643689 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.643693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:43.644350 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:43.644360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.644858 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.644861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:43.644962 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:43.645487 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:43.645498 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:43.646765 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:43.648752 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:43.648758 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:43.648761 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:43.648974 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:43.649281 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:43.649284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:43.649287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:43.649371 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:43.649851 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:43.649853 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:43.649930 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 06:49:43.650008 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:49:43.650177 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 06:49:43.650605 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.642133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45608 (local address 127.7.48.61:44341)
0928 06:49:43.642260 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:43.642263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:43.642275 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:49:43.642315 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:43.642317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:43.642356 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:43.642449 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:43.642892 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.643560 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.644494 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.644732 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.644814 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:43.646909 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:43.646913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:43.646915 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:43.646952 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:43.648650 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.649073 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.649078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.649080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.649163 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.649468 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.649471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.649473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.649738 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:43.649861 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:43.650097 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 06:49:43.650273 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:49:43.651396 30272 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:49:43.653522 30125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.642940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47494 (local address 127.7.48.62:33303)
0928 06:49:43.643047 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:43.643051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:43.643064 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:49:43.643118 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:43.643121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:43.643160 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:43.643237 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:43.643798 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.644326 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.645293 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.645573 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.645660 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:43.648275 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:43.648279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:43.648282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:43.648310 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:43.649983 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.650949 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.650951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.650952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.651054 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.652553 (+ 1499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.652556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.652558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.652783 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:43.653007 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:43.653209 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 06:49:43.653328 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:49:43.653512 30196 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } attempt: 3
Sep 28 06:49:43 dist-test-slave-twl6 krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578183, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.62@KRBTEST.COM for kudu/127.7.48.60@KRBTEST.COM
Sep 28 06:49:43 dist-test-slave-twl6 krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578183, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.61@KRBTEST.COM for kudu/127.7.48.60@KRBTEST.COM
Sep 28 06:49:43 dist-test-slave-twl6 krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.48.61@KRBTEST.COM
Sep 28 06:49:43 dist-test-slave-twl6 krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790578183, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.48.60@KRBTEST.COM for kudu/127.7.48.60@KRBTEST.COM
I20260928 06:49:43.664268 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.650562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:39611 (local address 127.0.0.1:54308)
0928 06:49:43.650757 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.650773 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:43.650848 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:43.651107 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:43.651110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:43.651299 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:43.651429 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:43.651437 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.652426 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.652430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:43.653224 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:43.653237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.653913 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.653916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:43.654020 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:43.654501 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:43.654515 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:43.656199 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:43.662276 (+ 6077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:43.662281 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:43.662288 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:43.662490 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:43.662805 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:43.662809 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:43.662811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:43.662924 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:43.663632 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:43.663635 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:43.663770 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 06:49:43.663882 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 06:49:43.664075 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260928 06:49:43.664831 30275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.650603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54308 (local address 127.7.48.60:39611)
0928 06:49:43.650904 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:43.650909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:43.650921 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:49:43.650952 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:43.650954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:43.651000 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:43.651079 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:43.651562 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.652335 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.653357 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.653755 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.655070 (+ 1315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:43.656875 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:43.656880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:43.656883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:43.656906 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:43.662066 (+ 5160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.662596 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.662602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.662607 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.662691 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.663276 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.663280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.663281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.663539 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:43.663635 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:43.664496 (+ 861us) server_negotiation.cc:299] Negotiation successful
0928 06:49:43.664634 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233}
I20260928 06:49:43.666338 30198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.653876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:39611 (local address 127.0.0.1:54330)
0928 06:49:43.653979 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.653994 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:43.654072 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:43.655020 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:43.655026 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:43.655236 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:43.655347 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:43.655354 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.656467 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.656472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:43.657151 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:43.657160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.657764 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.657768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:43.657847 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:43.660835 (+ 2988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:43.660853 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:43.662577 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:43.664646 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:43.664650 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:43.664652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:43.664931 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:43.665286 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:43.665289 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:43.665291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:43.665380 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:43.665848 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:43.665851 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:43.665944 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 06:49:43.666028 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 06:49:43.666175 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 06:49:43.666990 30278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.654393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54330 (local address 127.7.48.60:39611)
0928 06:49:43.654782 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:43.654792 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:43.654805 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:49:43.654836 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:43.654839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:43.654875 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:43.655490 (+ 615us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:43.655625 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.656358 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.657278 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.657647 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.658053 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:43.662706 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:43.662712 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:43.662718 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:43.662746 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:43.664522 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.665052 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.665055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.665056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.665168 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.665492 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.665495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.665496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.665735 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:43.666607 (+ 872us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:43.666682 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 06:49:43.666814 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"spinlock_wait_cycles":5888,"thread_start_us":98,"threads_started":1}
I20260928 06:49:43.668268 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.655357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45610 (local address 127.7.48.61:44341)
0928 06:49:43.655457 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:43.655463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:43.655477 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:49:43.655506 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:43.655508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:43.655551 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:43.655641 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:43.656850 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.657528 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.658476 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.658684 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.658766 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:43.664056 (+ 5290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:43.664060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:43.664062 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:43.664088 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:43.665874 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.666697 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.666703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.666706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.666794 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.667350 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.667357 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.667359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.667550 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:43.667766 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:43.668003 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 06:49:43.668120 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:49:43.668519 30121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9: Bootstrap starting.
I20260928 06:49:43.668628 30276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.651905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.48.60:39611 (local address 127.0.0.1:54316)
0928 06:49:43.652050 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 06:49:43.652063 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:49:43.652128 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:49:43.653082 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:49:43.653085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:49:43.653246 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:49:43.653350 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:43.653356 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.654935 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.654941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:43.656566 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:49:43.656578 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.657291 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.657298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:49:43.657396 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:43.661495 (+ 4099us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:49:43.661512 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:49:43.664220 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:49:43.667150 (+ 2930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:43.667153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:43.667155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:43.667396 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:43.667702 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:49:43.667704 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:49:43.667706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:49:43.667782 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:49:43.668216 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:49:43.668219 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:49:43.668292 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 06:49:43.668373 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 06:49:43.668490 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 06:49:43.669174 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.652015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54316 (local address 127.7.48.60:39611)
0928 06:49:43.652344 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:43.652353 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:43.652364 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:49:43.652398 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:43.652404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:43.652447 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:43.652539 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:43.653468 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.654814 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.656688 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.657071 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.657178 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:43.665113 (+ 7935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:43.665119 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:43.665121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:43.665162 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:43.667039 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.667508 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.667511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.667513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.667592 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.667896 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.667899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.667901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.668098 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:43.668185 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:43.668890 (+ 705us) server_negotiation.cc:299] Negotiation successful
0928 06:49:43.669012 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"spinlock_wait_cycles":1408,"thread_start_us":126,"threads_started":1}
I20260928 06:49:43.670702 30196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 695ca52f54024cbf9d32b2bd54fccaf9: Bootstrap starting.
I20260928 06:49:43.671490 30121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:43.671722 30272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a21b882a294a98a63adcb4e2b3368b: Bootstrap starting.
I20260928 06:49:43.671906 7360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:49:43.672468 30121 log.cc:824] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:43.673961 30196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 695ca52f54024cbf9d32b2bd54fccaf9: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:43.674070 30272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3a21b882a294a98a63adcb4e2b3368b: Neither blocks nor log segments found. Creating new log.
I20260928 06:49:43.673161 30095 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47494
I20260928 06:49:43.674674 30121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9: No bootstrap required, opened a new log
I20260928 06:49:43.674986 30272 log.cc:824] T 00000000000000000000000000000000 P d3a21b882a294a98a63adcb4e2b3368b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:43.675127 30196 log.cc:824] T 00000000000000000000000000000000 P 695ca52f54024cbf9d32b2bd54fccaf9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:49:43.677218 30272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a21b882a294a98a63adcb4e2b3368b: No bootstrap required, opened a new log
I20260928 06:49:43.677222 30196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 695ca52f54024cbf9d32b2bd54fccaf9: No bootstrap required, opened a new log
I20260928 06:49:43.677734 30121 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33303 } } peers { permanent_uuid: "695ca52f54024cbf9d32b2bd54fccaf9" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 44341 } } peers { permanent_uuid: "d3a21b882a294a98a63adcb4e2b3368b" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } }
I20260928 06:49:43.677963 30121 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b71d36de6b8455290a0b9e0e2bf14e9, State: Initialized, Role: FOLLOWER
I20260928 06:49:43.678297 30121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33303 } } peers { permanent_uuid: "695ca52f54024cbf9d32b2bd54fccaf9" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 44341 } } peers { permanent_uuid: "d3a21b882a294a98a63adcb4e2b3368b" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } }
I20260928 06:49:43.679697 30121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:43.679783 30095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 06:49:43.679981 30095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:43.681396 30095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33303 } } peers { permanent_uuid: "695ca52f54024cbf9d32b2bd54fccaf9" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 44341 } } peers { permanent_uuid: "d3a21b882a294a98a63adcb4e2b3368b" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } }
I20260928 06:49:43.681319 30196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 695ca52f54024cbf9d32b2bd54fccaf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33303 } } peers { permanent_uuid: "695ca52f54024cbf9d32b2bd54fccaf9" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 44341 } } peers { permanent_uuid: "d3a21b882a294a98a63adcb4e2b3368b" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } }
I20260928 06:49:43.681581 30196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 695ca52f54024cbf9d32b2bd54fccaf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 695ca52f54024cbf9d32b2bd54fccaf9, State: Initialized, Role: FOLLOWER
I20260928 06:49:43.681902 30196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 695ca52f54024cbf9d32b2bd54fccaf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33303 } } peers { permanent_uuid: "695ca52f54024cbf9d32b2bd54fccaf9" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 44341 } } peers { permanent_uuid: "d3a21b882a294a98a63adcb4e2b3368b" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } }
I20260928 06:49:43.681965 30272 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3a21b882a294a98a63adcb4e2b3368b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33303 } } peers { permanent_uuid: "695ca52f54024cbf9d32b2bd54fccaf9" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 44341 } } peers { permanent_uuid: "d3a21b882a294a98a63adcb4e2b3368b" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } }
I20260928 06:49:43.682207 30272 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3a21b882a294a98a63adcb4e2b3368b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a21b882a294a98a63adcb4e2b3368b, State: Initialized, Role: FOLLOWER
I20260928 06:49:43.682220 30282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33303 } } peers { permanent_uuid: "695ca52f54024cbf9d32b2bd54fccaf9" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 44341 } } peers { permanent_uuid: "d3a21b882a294a98a63adcb4e2b3368b" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } } }
I20260928 06:49:43.682336 30282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:49:43.682490 30272 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3a21b882a294a98a63adcb4e2b3368b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33303 } } peers { permanent_uuid: "695ca52f54024cbf9d32b2bd54fccaf9" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 44341 } } peers { permanent_uuid: "d3a21b882a294a98a63adcb4e2b3368b" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } }
I20260928 06:49:43.682883 30095 leader_election.cc:288] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9 [CANDIDATE]: Term 1 election: Requested vote from peers 695ca52f54024cbf9d32b2bd54fccaf9 (127.7.48.61:44341), d3a21b882a294a98a63adcb4e2b3368b (127.7.48.60:39611)
I20260928 06:49:43.683327 30196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 695ca52f54024cbf9d32b2bd54fccaf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:43.683456 30169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "695ca52f54024cbf9d32b2bd54fccaf9"
I20260928 06:49:43.683670 30169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 695ca52f54024cbf9d32b2bd54fccaf9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:43.683770 30272 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3a21b882a294a98a63adcb4e2b3368b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:49:43.685675 30169 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 695ca52f54024cbf9d32b2bd54fccaf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b71d36de6b8455290a0b9e0e2bf14e9 in term 1.
I20260928 06:49:43.685993 30246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3a21b882a294a98a63adcb4e2b3368b"
I20260928 06:49:43.686820 30070 leader_election.cc:302] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b71d36de6b8455290a0b9e0e2bf14e9, 695ca52f54024cbf9d32b2bd54fccaf9; no voters:
I20260928 06:49:43.686260 30246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3a21b882a294a98a63adcb4e2b3368b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:49:43.687853 30282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:49:43.688302 30282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9 [term 1 LEADER]: Becoming Leader. State: Replica: 5b71d36de6b8455290a0b9e0e2bf14e9, State: Running, Role: LEADER
I20260928 06:49:43.688361 30246 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d3a21b882a294a98a63adcb4e2b3368b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b71d36de6b8455290a0b9e0e2bf14e9 in term 1.
I20260928 06:49:43.687074 30293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 695ca52f54024cbf9d32b2bd54fccaf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33303 } } peers { permanent_uuid: "695ca52f54024cbf9d32b2bd54fccaf9" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 44341 } } peers { permanent_uuid: "d3a21b882a294a98a63adcb4e2b3368b" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } } }
I20260928 06:49:43.688628 30294 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a21b882a294a98a63adcb4e2b3368b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33303 } } peers { permanent_uuid: "695ca52f54024cbf9d32b2bd54fccaf9" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 44341 } } peers { permanent_uuid: "d3a21b882a294a98a63adcb4e2b3368b" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } } }
I20260928 06:49:43.688740 30294 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a21b882a294a98a63adcb4e2b3368b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:49:43.688669 30282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33303 } } peers { permanent_uuid: "695ca52f54024cbf9d32b2bd54fccaf9" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 44341 } } peers { permanent_uuid: "d3a21b882a294a98a63adcb4e2b3368b" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } }
I20260928 06:49:43.688741 30293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 695ca52f54024cbf9d32b2bd54fccaf9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:49:43.690557 30282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b71d36de6b8455290a0b9e0e2bf14e9. Latest consensus state: current_term: 1 leader_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33303 } } peers { permanent_uuid: "695ca52f54024cbf9d32b2bd54fccaf9" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 44341 } } peers { permanent_uuid: "d3a21b882a294a98a63adcb4e2b3368b" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } } }
I20260928 06:49:43.690649 30282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9 [sys.catalog]: This master's current role is: LEADER
W20260928 06:49:43.692216 30322 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 695ca52f54024cbf9d32b2bd54fccaf9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:49:43.692225 30320 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3a21b882a294a98a63adcb4e2b3368b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:49:43.692292 30320 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 06:49:43.692296 30322 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 06:49:43.695865 30318 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:49:43.695973 30318 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:49:43.696695 30329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:49:43.697708 30329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:49:43.700073 30204 master_runner.cc:263] No leader master found from master d3a21b882a294a98a63adcb4e2b3368b
I20260928 06:49:43.700147 30204 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:49:43.700937 30053 master_runner.cc:263] No leader master found from master 5b71d36de6b8455290a0b9e0e2bf14e9
I20260928 06:49:43.701016 30053 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:49:43.706014 30125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.688437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47498 (local address 127.7.48.62:33303)
0928 06:49:43.688545 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:43.688550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:43.692598 (+ 4048us) server_negotiation.cc:407] Connection header received
0928 06:49:43.692701 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:43.692705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:43.692754 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:43.692826 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:43.695558 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.696294 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.697429 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.697766 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.697838 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:43.702317 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:43.702321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:43.702324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:43.702354 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:43.704020 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.704518 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.704522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.704524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.704614 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.704942 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.704945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.704947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.705189 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:43.705312 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:43.705698 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 06:49:43.705832 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:49:43.706283 30275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.686244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54346 (local address 127.7.48.60:39611)
0928 06:49:43.686780 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:43.686786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:43.689274 (+ 2488us) server_negotiation.cc:407] Connection header received
0928 06:49:43.689382 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:43.689386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:43.689436 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:43.689517 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:43.690484 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.691221 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.692381 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.693029 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.693124 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:43.702166 (+ 9042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:43.702171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:43.702174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:43.702201 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:43.704395 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.704897 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.704901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.704903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.704989 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.705322 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.705326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.705328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.705546 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:43.705658 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:43.705997 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 06:49:43.706129 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438}
I20260928 06:49:43.713107 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:43.687987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45614 (local address 127.7.48.61:44341)
0928 06:49:43.688651 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:43.688657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:43.696250 (+ 7593us) server_negotiation.cc:407] Connection header received
0928 06:49:43.696427 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:43.696431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:43.696476 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:43.696526 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:43.697430 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.698123 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.699206 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:43.699545 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:43.699644 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:43.708809 (+ 9165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:49:43.708814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:49:43.708817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:49:43.708851 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:49:43.710595 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.711151 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.711155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.711157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.711292 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:49:43.711629 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:49:43.711632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:49:43.711634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:49:43.711872 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:49:43.712035 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:43.712797 (+ 762us) server_negotiation.cc:299] Negotiation successful
0928 06:49:43.712925 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588}
I20260928 06:49:43.713415 30169 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 695ca52f54024cbf9d32b2bd54fccaf9 [term 1 FOLLOWER]: Refusing update from remote peer 5b71d36de6b8455290a0b9e0e2bf14e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:49:43.714234 30325 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "695ca52f54024cbf9d32b2bd54fccaf9" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 44341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:49:43.714716 30246 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d3a21b882a294a98a63adcb4e2b3368b [term 1 FOLLOWER]: Refusing update from remote peer 5b71d36de6b8455290a0b9e0e2bf14e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:49:43.715449 30325 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a21b882a294a98a63adcb4e2b3368b" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:49:43.718705 30293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 695ca52f54024cbf9d32b2bd54fccaf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b71d36de6b8455290a0b9e0e2bf14e9. Latest consensus state: current_term: 1 leader_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33303 } } peers { permanent_uuid: "695ca52f54024cbf9d32b2bd54fccaf9" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 44341 } } peers { permanent_uuid: "d3a21b882a294a98a63adcb4e2b3368b" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } } }
I20260928 06:49:43.718814 30293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 695ca52f54024cbf9d32b2bd54fccaf9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:49:43.719477 30294 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a21b882a294a98a63adcb4e2b3368b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b71d36de6b8455290a0b9e0e2bf14e9. Latest consensus state: current_term: 1 leader_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33303 } } peers { permanent_uuid: "695ca52f54024cbf9d32b2bd54fccaf9" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 44341 } } peers { permanent_uuid: "d3a21b882a294a98a63adcb4e2b3368b" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } } }
I20260928 06:49:43.719609 30294 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a21b882a294a98a63adcb4e2b3368b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:49:43.725296 30282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33303 } } peers { permanent_uuid: "695ca52f54024cbf9d32b2bd54fccaf9" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 44341 } } peers { permanent_uuid: "d3a21b882a294a98a63adcb4e2b3368b" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } } }
I20260928 06:49:43.725430 30282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:43.725522 30282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33303 } } peers { permanent_uuid: "695ca52f54024cbf9d32b2bd54fccaf9" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 44341 } } peers { permanent_uuid: "d3a21b882a294a98a63adcb4e2b3368b" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } } }
I20260928 06:49:43.725581 30282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9 [sys.catalog]: This master's current role is: LEADER
I20260928 06:49:43.725612 30293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 695ca52f54024cbf9d32b2bd54fccaf9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33303 } } peers { permanent_uuid: "695ca52f54024cbf9d32b2bd54fccaf9" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 44341 } } peers { permanent_uuid: "d3a21b882a294a98a63adcb4e2b3368b" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } } }
I20260928 06:49:43.725675 30331 mvcc.cc:205] Tried to move back new op lower bound from 7334208240480468992 to 7334208240392552448. Current Snapshot: MvccSnapshot[applied={T|T < 7334208240480468992}]
I20260928 06:49:43.725730 30293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 695ca52f54024cbf9d32b2bd54fccaf9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:49:43.725824 30294 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a21b882a294a98a63adcb4e2b3368b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b71d36de6b8455290a0b9e0e2bf14e9" member_type: VOTER last_known_addr { host: "127.7.48.62" port: 33303 } } peers { permanent_uuid: "695ca52f54024cbf9d32b2bd54fccaf9" member_type: VOTER last_known_addr { host: "127.7.48.61" port: 44341 } } peers { permanent_uuid: "d3a21b882a294a98a63adcb4e2b3368b" member_type: VOTER last_known_addr { host: "127.7.48.60" port: 39611 } } }
I20260928 06:49:43.725939 30294 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a21b882a294a98a63adcb4e2b3368b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:49:43.726253 30334 mvcc.cc:205] Tried to move back new op lower bound from 7334208240480468992 to 7334208240392552448. Current Snapshot: MvccSnapshot[applied={T|T < 7334208240480468992}]
I20260928 06:49:43.726538 30332 mvcc.cc:205] Tried to move back new op lower bound from 7334208240480468992 to 7334208240392552448. Current Snapshot: MvccSnapshot[applied={T|T < 7334208240480468992}]
I20260928 06:49:43.736002 30329 catalog_manager.cc:1347] Generated new cluster ID: 6e01f1529a4c4573ae593a31b9da5a35
I20260928 06:49:43.736094 30329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:49:43.780306 30329 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:49:43.783946 30329 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:49:43.809455 30329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b71d36de6b8455290a0b9e0e2bf14e9: Generated new TSK 0
I20260928 06:49:43.810514 30329 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:49:43.842914 30275 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:43.840333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54354 (local address 127.7.48.60:39611)
0928 06:49:43.840465 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:43.840470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:43.841795 (+ 1325us) server_negotiation.cc:407] Connection header received
0928 06:49:43.841991 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:43.841995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:43.842057 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:43.842110 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:43.842775 (+ 665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54354: BlockingRecv error: recv got EOF from 127.0.0.1:54354 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 06:49:43.844547 30202 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:43.841178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45624 (local address 127.7.48.61:44341)
0928 06:49:43.841289 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:43.841293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:43.843574 (+ 2281us) server_negotiation.cc:407] Connection header received
0928 06:49:43.843742 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:43.843746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:43.843797 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:43.843847 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:43.844425 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45624: BlockingRecv error: recv got EOF from 127.0.0.1:45624 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 06:49:43.846585 30125 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:43.845340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47504 (local address 127.7.48.62:33303)
0928 06:49:43.845488 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:43.845493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:43.845687 (+ 194us) server_negotiation.cc:407] Connection header received
0928 06:49:43.845858 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:43.845861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:43.845911 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:43.845965 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:43.846471 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47504: BlockingRecv error: recv got EOF from 127.0.0.1:47504 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 06:49:43.851238 30202 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:43.849653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45626 (local address 127.7.48.61:44341)
0928 06:49:43.849765 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:43.849769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:43.850245 (+ 476us) server_negotiation.cc:407] Connection header received
0928 06:49:43.850487 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:43.850491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:43.850568 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:43.850623 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:43.851138 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45626: BlockingRecv error: recv got EOF from 127.0.0.1:45626 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 06:49:43.856943 30202 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:43.855236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45632 (local address 127.7.48.61:44341)
0928 06:49:43.855346 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:43.855350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:43.855965 (+ 615us) server_negotiation.cc:407] Connection header received
0928 06:49:43.856190 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:43.856196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:43.856251 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:43.856304 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:43.856835 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45632: BlockingRecv error: recv got EOF from 127.0.0.1:45632 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 06:49:43.863128 30202 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:43.861488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45646 (local address 127.7.48.61:44341)
0928 06:49:43.861594 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:43.861598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:43.862159 (+ 561us) server_negotiation.cc:407] Connection header received
0928 06:49:43.862352 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:43.862356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:43.862406 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:43.862460 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:43.863000 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45646: BlockingRecv error: recv got EOF from 127.0.0.1:45646 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 06:49:43.872944 30202 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:43.870824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45656 (local address 127.7.48.61:44341)
0928 06:49:43.870999 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:43.871004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:43.871628 (+ 624us) server_negotiation.cc:407] Connection header received
0928 06:49:43.871863 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:43.871867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:43.871990 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:43.872079 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:43.872818 (+ 739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45656: BlockingRecv error: recv got EOF from 127.0.0.1:45656 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260928 06:49:43.886509 30202 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:43.884241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45662 (local address 127.7.48.61:44341)
0928 06:49:43.884390 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:43.884395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:43.885292 (+ 897us) server_negotiation.cc:407] Connection header received
0928 06:49:43.885617 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:43.885621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:43.885685 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:43.885746 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:43.886382 (+ 636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45662: BlockingRecv error: recv got EOF from 127.0.0.1:45662 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 06:49:43.907284 30202 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:43.905434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45670 (local address 127.7.48.61:44341)
0928 06:49:43.905569 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:43.905576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:43.906141 (+ 565us) server_negotiation.cc:407] Connection header received
0928 06:49:43.906401 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:43.906407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:43.906481 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:43.906546 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:43.907157 (+ 611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45670: BlockingRecv error: recv got EOF from 127.0.0.1:45670 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 06:49:43.943352 30202 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:43.942139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45678 (local address 127.7.48.61:44341)
0928 06:49:43.942292 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:43.942297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:43.942475 (+ 178us) server_negotiation.cc:407] Connection header received
0928 06:49:43.942594 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:43.942597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:43.942647 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:43.942721 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:43.943201 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45678: BlockingRecv error: recv got EOF from 127.0.0.1:45678 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 06:49:44.011267 30202 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:44.009735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45682 (local address 127.7.48.61:44341)
0928 06:49:44.010013 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:44.010018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:44.010169 (+ 151us) server_negotiation.cc:407] Connection header received
0928 06:49:44.010362 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:44.010371 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:44.010420 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:44.010520 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:44.011134 (+ 614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45682: BlockingRecv error: recv got EOF from 127.0.0.1:45682 (error 108)
Metrics: {"server-negotiator.queue_time_us":168}
W20260928 06:49:44.142649 30202 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:44.141592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45684 (local address 127.7.48.61:44341)
0928 06:49:44.141740 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:44.141744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:44.141863 (+ 119us) server_negotiation.cc:407] Connection header received
0928 06:49:44.141991 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:44.141994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:44.142042 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:44.142125 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:44.142536 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45684: BlockingRecv error: recv got EOF from 127.0.0.1:45684 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 06:49:44.402314 30202 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:49:44.401079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45692 (local address 127.7.48.61:44341)
0928 06:49:44.401235 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:44.401239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:44.401377 (+ 138us) server_negotiation.cc:407] Connection header received
0928 06:49:44.401529 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:44.401531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:44.401578 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:44.401646 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:49:44.402182 (+ 536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45692: BlockingRecv error: recv got EOF from 127.0.0.1:45692 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 06:49:44.693598 30322 catalog_manager.cc:1259] Loaded cluster ID: 6e01f1529a4c4573ae593a31b9da5a35
I20260928 06:49:44.693591 30320 catalog_manager.cc:1259] Loaded cluster ID: 6e01f1529a4c4573ae593a31b9da5a35
I20260928 06:49:44.693717 30320 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d3a21b882a294a98a63adcb4e2b3368b: loading cluster ID for follower catalog manager: success
I20260928 06:49:44.693719 30322 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 695ca52f54024cbf9d32b2bd54fccaf9: loading cluster ID for follower catalog manager: success
I20260928 06:49:44.695344 30322 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 695ca52f54024cbf9d32b2bd54fccaf9: acquiring CA information for follower catalog manager: success
I20260928 06:49:44.695345 30320 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d3a21b882a294a98a63adcb4e2b3368b: acquiring CA information for follower catalog manager: success
I20260928 06:49:44.696604 30322 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 695ca52f54024cbf9d32b2bd54fccaf9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:49:44.696622 30320 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d3a21b882a294a98a63adcb4e2b3368b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:49:44.920143 30404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:44.916470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42984 (local address 127.7.48.61:44341)
0928 06:49:44.916776 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:44.916779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:44.916795 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:49:44.916895 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:44.916897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:44.917020 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:44.917121 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:49:44.917367 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:44.917894 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:44.918731 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:44.918948 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:44.919024 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:44.919299 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:49:44.919753 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:49:44.919847 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:44.919962 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 06:49:44.920014 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":108,"threads_started":1}
I20260928 06:49:44.926604 30410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:44.923174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33914 (local address 127.7.48.60:39611)
0928 06:49:44.923383 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:44.923387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:44.923400 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:49:44.923555 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:44.923557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:44.923677 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:44.923780 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:49:44.924014 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:44.924520 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:44.925285 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:44.925497 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:44.925571 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:44.925814 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:49:44.926240 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:49:44.926332 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:44.926434 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 06:49:44.926483 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":67,"threads_started":1}
I20260928 06:49:44.933370 30404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:44.929761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42992 (local address 127.7.48.61:44341)
0928 06:49:44.929908 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:44.929913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:44.929957 (+ 44us) server_negotiation.cc:407] Connection header received
0928 06:49:44.930009 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:44.930012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:44.930074 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:44.930163 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:49:44.930353 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:44.931018 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:44.932051 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:44.932325 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:44.932435 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:44.932826 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:49:44.932905 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:49:44.932948 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:44.933130 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 06:49:44.933196 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:49:44.933969 30410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:44.929890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33916 (local address 127.7.48.60:39611)
0928 06:49:44.930331 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:44.930338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:44.930353 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:49:44.930467 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:44.930470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:44.930514 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:44.930584 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:49:44.930829 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:44.931537 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:44.932733 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:44.933047 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:44.933152 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:44.933541 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:49:44.933613 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:49:44.933673 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:44.933805 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 06:49:44.933858 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359}
I20260928 06:49:44.935079 30417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:49:44.929629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33164 (local address 127.7.48.62:33303)
0928 06:49:44.930604 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0928 06:49:44.930609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:49:44.930631 (+ 22us) server_negotiation.cc:407] Connection header received
0928 06:49:44.930732 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:49:44.930734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:49:44.930912 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:49:44.931036 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:49:44.931353 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:44.932065 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:44.933387 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:49:44.933772 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:49:44.933863 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:49:44.934430 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:49:44.934740 (+ 310us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:49:44.934816 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:49:44.934921 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 06:49:44.934971 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":889,"thread_start_us":78,"threads_started":1}
I20260928 06:49:44.938047 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 30053
I20260928 06:49:44.945236 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 30128
I20260928 06:49:44.951406 7360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskipM9PC/build/debug/bin/kudu with pid 30204
2026-09-28T06:49:44Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3100 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6196 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (279418 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST